# E2E Test: Full AI Support Automation Workflow

## System Setup Prerequisites

### Services Health Check
- **Milvus Vector DB**: Accessible on port 9091
- **PostgreSQL**: Running with KB schema and audit tables
- **n8n**: Workflow engine operational
- **Freescout**: Help desk system with custom fields configured
- **LiteLLM**: LLM proxy service accessible
- **Baramundi**: Remote execution system (optional)

### Test Environment Variables
```bash
export FREESCOUT_API_KEY="your-api-key"
export LITELLM_API_KEY="your-api-key"
export N8N_AUTH_TOKEN="your-auth-token"
export POSTGRES_PASSWORD="your-password"
```

---

## E2E Test Scenario: Complete Workflow

### Test Case ID: E2E-001
**Objective**: Full workflow from ticket creation through knowledge base update

---

## 1. Setup Phase

### Pre-Test Validation
```bash
# Verify all services are running
./tests/curl-test-collection.sh

# Clear test data from previous runs (optional)
docker-compose exec postgres psql -U kb_user -d n8n_kb -c \
  "DELETE FROM knowledge_base_updates WHERE created_at > NOW() - INTERVAL '1 hour';"

docker-compose exec postgres psql -U kb_user -d n8n_kb -c \
  "DELETE FROM ticket_audit WHERE ticket_id LIKE 'TEST_%';"
```

### Test Data Preparation
- Create test customer: `test@example.com` in Freescout (if not exists)
- Verify custom fields exist in Freescout:
  - `AI_SUGGESTION` (text field)
  - `AI_SUGGESTION_STATUS` (enum: PENDING, APPROVED, REJECTED, EXECUTED)
  - `AI_CONFIDENCE` (number field)

---

## 2. Workflow A: Ticket Analysis & AI Suggestion

### Test Step 1: Create Test Ticket in Freescout

**Action**: Create new ticket via Freescout UI or API
```bash
curl -X POST https://ekshelpdesk.fft-it.de/api/v1/mailboxes/1/conversations \
  -H "Authorization: Bearer $FREESCOUT_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "subject": "TEST_E2E_001: Drucker funktioniert nicht",
    "body": "Jeder Druck-Befehl wird abgelehnt. Fehlercode 5. Bitte schnell lösen!",
    "customer_email": "test@example.com",
    "mailbox_id": 1
  }'
```

**Expected Outcome**:
- HTTP Status: 201 (Created)
- Response contains `conversation_id`
- Ticket visible in Freescout dashboard

**Validation**:
```bash
# Get the created ticket ID from response
TICKET_ID="<conversation_id_from_response>"
echo "Created Ticket: $TICKET_ID"
```

---

### Test Step 2: Wait for Workflow A Cycle (5 minutes)

**Process Flow**:
1. n8n Workflow A triggers every 5 minutes
2. Fetches new tickets from Freescout
3. Sends ticket text to LiteLLM for analysis
4. Updates custom fields with AI suggestion

**Wait Time**: 5 minutes (plus 1 min buffer = 6 minutes total)

**During Wait - Monitor Logs**:
```bash
# Monitor n8n logs for workflow execution
docker-compose logs -f n8n | grep -i "workflow_a\|ticket\|analysis"

# Check PostgreSQL audit log
docker-compose exec postgres psql -U kb_user -d n8n_kb -c \
  "SELECT * FROM ticket_audit WHERE ticket_id = '$TICKET_ID' ORDER BY created_at DESC;"
```

**Expected n8n Logs**:
- "Processing ticket: TEST_E2E_001"
- "Calling LiteLLM API for analysis"
- "Analysis complete: category=Hardware, confidence=0.92"
- "Updated custom fields in Freescout"

---

### Test Step 3: Verify AI Suggestion in Freescout

**Action**: Check ticket custom fields
```bash
curl -H "Authorization: Bearer $FREESCOUT_API_KEY" \
  https://ekshelpdesk.fft-it.de/api/v1/conversations/$TICKET_ID
```

**Expected Response Fields**:
```json
{
  "conversation": {
    "id": "TICKET_ID",
    "subject": "TEST_E2E_001: Drucker funktioniert nicht",
    "custom_fields": {
      "AI_SUGGESTION": "Hardware Problem: Drucker-Treiber fehlerhaft oder beschädigt. Empfohlene Lösung: 1) Drucker neustarten, 2) Treiber neu installieren",
      "AI_SUGGESTION_STATUS": "PENDING",
      "AI_CONFIDENCE": 0.92
    }
  }
}
```

**Validation Checklist**:
- ✅ `AI_SUGGESTION` is not empty
- ✅ `AI_SUGGESTION_STATUS` = "PENDING"
- ✅ `AI_CONFIDENCE` between 0.7 and 1.0
- ✅ Suggestion text contains problem category and solution

**Alternative: Manual Check in UI**:
- Open ticket in Freescout
- Scroll to custom fields section
- Verify all three fields populated with expected values

---

## 3. Workflow B: Approval & Execution

### Test Step 4: Approve AI Suggestion

**Action**: Update custom field via API
```bash
curl -X PUT https://ekshelpdesk.fft-it.de/api/v1/conversations/$TICKET_ID \
  -H "Authorization: Bearer $FREESCOUT_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "custom_fields": {
      "AI_SUGGESTION_STATUS": "APPROVED"
    }
  }'
```

**Expected Outcome**:
- HTTP Status: 200 (OK)
- Custom field updated in Freescout

**Validation**:
```bash
# Verify field was updated
curl -H "Authorization: Bearer $FREESCOUT_API_KEY" \
  https://ekshelpdesk.fft-it.de/api/v1/conversations/$TICKET_ID | \
  jq '.conversation.custom_fields.AI_SUGGESTION_STATUS'
# Expected output: "APPROVED"
```

---

### Test Step 5: Wait for Workflow B Cycle (2 minutes)

**Process Flow**:
1. n8n Workflow B triggers every 2 minutes
2. Fetches tickets with `AI_SUGGESTION_STATUS = APPROVED`
3. Executes action based on category:
   - **Hardware/Remote**: Create Baramundi job
   - **Software/Config**: Send support email
   - **Knowledge**: Update KB directly
4. Updates status to `EXECUTED`

**Wait Time**: 2 minutes (plus 1 min buffer = 3 minutes total)

**During Wait - Monitor Logs**:
```bash
# Check n8n workflow B execution
docker-compose logs -f n8n | grep -i "workflow_b\|approved\|executing"

# Check for Baramundi job creation (if applicable)
curl https://baramundi-api.example.com/jobs \
  -H "Authorization: Bearer $BARAMUNDI_TOKEN" | \
  jq '.jobs[] | select(.ticket_id == "'"$TICKET_ID"'")'
```

---

### Test Step 6: Verify Execution Status

**Action**: Check ticket status
```bash
curl -H "Authorization: Bearer $FREESCOUT_API_KEY" \
  https://ekshelpdesk.fft-it.de/api/v1/conversations/$TICKET_ID
```

**Expected Response**:
```json
{
  "conversation": {
    "custom_fields": {
      "AI_SUGGESTION_STATUS": "EXECUTED",
      "EXECUTION_TIMESTAMP": "2026-03-16T14:35:00Z"
    }
  }
}
```

**Validation Checklist**:
- ✅ `AI_SUGGESTION_STATUS` = "EXECUTED"
- ✅ `EXECUTION_TIMESTAMP` is recent
- ✅ No errors in n8n logs

---

## 4. Workflow C: Knowledge Base Update

### Test Step 7: Wait for Workflow C Cycle (1 minute)

**Process Flow**:
1. n8n Workflow C triggers every 1 minute
2. Fetches executed tickets
3. Extracts: Problem, Solution, Category
4. Inserts into PostgreSQL `knowledge_base_updates` table
5. Triggers Milvus vector DB embedding and indexing

**Wait Time**: 1 minute (plus 1 min buffer = 2 minutes total)

---

### Test Step 8: Verify Knowledge Base Entry

**Action 1**: Check PostgreSQL insert
```bash
docker-compose exec postgres psql -U kb_user -d n8n_kb -c \
  "SELECT id, ticket_id, problem, solution, category, frequency, created_at
   FROM knowledge_base_updates
   WHERE ticket_id = '$TICKET_ID'
   ORDER BY created_at DESC LIMIT 1;"
```

**Expected Output**:
```
 id | ticket_id | problem | solution | category | frequency | created_at
----+-----------+---------+----------+----------+-----------+---------------------
 42 | TEST...   | Drucker | Treiber  | Hardware | 1         | 2026-03-16 14:35:00
```

**Action 2**: Check Milvus vector DB
```bash
# Query Milvus for the new KB entry
curl -X POST http://127.0.0.1:19530/v1/search \
  -H "Content-Type: application/json" \
  -d '{
    "collection_name": "knowledge_base",
    "vectors": [
      "vector_embedding_of_problem_text"
    ],
    "top_k": 10,
    "output_fields": ["id", "ticket_id", "problem", "solution", "similarity"]
  }' | jq '.'
```

**Expected Response**:
```json
{
  "results": [
    {
      "id": 42,
      "ticket_id": "TEST_E2E_001",
      "problem": "Drucker funktioniert nicht - Fehlercode 5",
      "solution": "Drucker neustarten und Treiber neu installieren",
      "similarity": 0.98
    }
  ]
}
```

**Validation Checklist**:
- ✅ Record exists in `knowledge_base_updates` table
- ✅ Fields populated: problem, solution, category, frequency
- ✅ Record appears in Milvus search results
- ✅ Similarity score >= 0.95 for exact match

---

## 5. Vector DB Search Test

### Test Step 9: Search Similar Problem

**Objective**: Test that similar problems can be found via vector similarity

**Action**: Query Milvus with similar but different text
```bash
# Prepare embedding for similar problem
curl -X POST http://127.0.0.1:19530/v1/search \
  -H "Content-Type: application/json" \
  -d '{
    "collection_name": "knowledge_base",
    "vectors": [
      "vector_for_query: Druckerprobleme beim Ausdrucken mit Fehler"
    ],
    "top_k": 5,
    "output_fields": ["id", "ticket_id", "problem", "solution", "category"]
  }'
```

**Expected Outcome**:
- Top result is our test KB entry
- Similarity score > 0.85
- Same category returned (Hardware)

**Expected Result**:
```json
{
  "results": [
    {
      "rank": 1,
      "similarity": 0.91,
      "ticket_id": "TEST_E2E_001",
      "problem": "Drucker funktioniert nicht - Fehlercode 5",
      "solution": "Drucker neustarten und Treiber neu installieren",
      "category": "Hardware"
    },
    {
      "rank": 2,
      "similarity": 0.78,
      "ticket_id": "OTHER_TICKET",
      "problem": "Netzwerkdrucker nicht erreichbar",
      "solution": "IP-Konfiguration prüfen"
    }
  ]
}
```

**Validation**:
- ✅ Test entry ranks in top 3
- ✅ Similarity > 0.85
- ✅ Relevant results returned

---

## 6. Failure Scenarios

### Scenario F1: Service Unavailable

**Test**: What happens if LiteLLM is unreachable?

**Action**: Stop LiteLLM service
```bash
docker-compose stop litellm
```

**Expected Behavior**:
- n8n Workflow A fails gracefully
- Error logged in PostgreSQL error_log table
- Ticket status remains PENDING
- After service recovery, retry happens automatically

**Verify**:
```bash
docker-compose exec postgres psql -U kb_user -d n8n_kb -c \
  "SELECT error_message, retry_count FROM error_log WHERE ticket_id = '$TICKET_ID';"
```

**Cleanup**: Restart service
```bash
docker-compose up -d litellm
```

---

### Scenario F2: Invalid Custom Field

**Test**: What if custom field value is invalid?

**Action**: Set invalid status value
```bash
curl -X PUT https://ekshelpdesk.fft-it.de/api/v1/conversations/$TICKET_ID \
  -H "Authorization: Bearer $FREESCOUT_API_KEY" \
  -d '{
    "custom_fields": {
      "AI_SUGGESTION_STATUS": "INVALID_VALUE"
    }
  }'
```

**Expected Behavior**:
- Workflow B ignores ticket
- Error logged
- Manual intervention required

---

## 7. Test Cleanup

### Post-Test Actions
```bash
# Archive test data
docker-compose exec postgres psql -U kb_user -d n8n_kb -c \
  "UPDATE ticket_audit
   SET archived = true
   WHERE ticket_id LIKE 'TEST_%';"

# Optional: Delete test ticket from Freescout
curl -X DELETE https://ekshelpdesk.fft-it.de/api/v1/conversations/$TICKET_ID \
  -H "Authorization: Bearer $FREESCOUT_API_KEY"

# Verify cleanup
docker-compose exec postgres psql -U kb_user -d n8n_kb -c \
  "SELECT COUNT(*) as active_test_tickets FROM ticket_audit
   WHERE ticket_id LIKE 'TEST_%' AND archived = false;"
```

---

## 8. Test Metrics & Success Criteria

### Timing Metrics
| Phase | Expected Duration | Tolerance |
|-------|------------------|-----------|
| Setup | - | - |
| Workflow A (Analysis) | 5 min | ±1 min |
| Workflow B (Execution) | 2 min | ±30 sec |
| Workflow C (KB Update) | 1 min | ±30 sec |
| **Total End-to-End** | **~8 min** | **±2 min** |

### Success Criteria
- ✅ All services respond to health checks
- ✅ Ticket created successfully in Freescout
- ✅ AI analysis completes within 5 min
- ✅ Confidence score >= 0.7
- ✅ Approval triggers execution within 2 min
- ✅ KB entry created in PostgreSQL
- ✅ KB entry indexed in Milvus
- ✅ Vector search returns relevant results (similarity > 0.85)
- ✅ All audit logs recorded correctly
- ✅ No unhandled errors in logs

### Performance Targets
- API response time: < 2 seconds
- LiteLLM inference: < 10 seconds
- Milvus embedding + indexing: < 5 seconds
- PostgreSQL inserts: < 1 second

---

## 9. Test Execution Checklist

```bash
# Start here:
[ ] Verify all services running: ./tests/curl-test-collection.sh
[ ] Create test ticket (capture TICKET_ID)
[ ] Wait 6 minutes for Workflow A
[ ] Verify AI_SUGGESTION populated
[ ] Approve ticket (update AI_SUGGESTION_STATUS)
[ ] Wait 3 minutes for Workflow B
[ ] Verify AI_SUGGESTION_STATUS = EXECUTED
[ ] Wait 2 minutes for Workflow C
[ ] Verify PostgreSQL kb entry exists
[ ] Verify Milvus vector search works
[ ] Review all audit logs
[ ] Clean up test data
[ ] Document any issues found
```

---

## 10. Troubleshooting Guide

### Issue: AI_SUGGESTION not populated after 5 min

**Check**:
1. n8n logs: `docker-compose logs n8n | grep -i error`
2. Freescout API connectivity: `curl https://ekshelpdesk.fft-it.de/healthz`
3. LiteLLM service: `curl http://llm.eks-ai.apps.asgard.eks-lnx.fft-it.de/health`
4. Custom field exists: Check Freescout admin panel

### Issue: Workflow B not executing

**Check**:
1. Ticket status field correct: `curl -H "Auth: Bearer $KEY" https://ekshelpdesk.fft-it.de/api/v1/conversations/$TICKET_ID | jq '.custom_fields.AI_SUGGESTION_STATUS'`
2. n8n Workflow B enabled: Check n8n UI
3. Error logs: `docker-compose exec postgres psql -U kb_user -d n8n_kb -c "SELECT * FROM error_log ORDER BY created_at DESC LIMIT 5;"`

### Issue: Milvus search returns no results

**Check**:
1. Milvus running: `curl http://127.0.0.1:19530/v1/healthz`
2. KB collection exists: See Milvus documentation for collection listing
3. Vector embedding generated: Check PostgreSQL `knowledge_base_updates.vector_embedding`

---

## Appendix A: API Endpoints Reference

| Service | Endpoint | Purpose |
|---------|----------|---------|
| Freescout | `https://ekshelpdesk.fft-it.de/api/v1/conversations` | Ticket management |
| LiteLLM | `http://llm.eks-ai.apps.asgard.eks-lnx.fft-it.de/v1/chat/completions` | AI analysis |
| Milvus | `http://127.0.0.1:19530/v1/search` | Vector DB search |
| PostgreSQL | `localhost:5432` | Data persistence |
| n8n | `http://localhost:5678` | Workflow orchestration |

---

**Document Version**: 1.0
**Last Updated**: 2026-03-16
**Author**: QA/Testing Agent
**Status**: Ready for Testing