test: E2E test scenarios and curl collection

2026-03-16 17:33:38 +01:00
parent caac04c684
commit fc4a947365
2 changed files with 1041 additions and 0 deletions
--- a/tests/e2e-test-scenario.md
+++ b/tests/e2e-test-scenario.md
@@ -0,0 +1,527 @@
+# E2E Test: Full AI Support Automation Workflow
+
+## System Setup Prerequisites
+
+### Services Health Check
+- **Milvus Vector DB**: Accessible on port 9091
+- **PostgreSQL**: Running with KB schema and audit tables
+- **n8n**: Workflow engine operational
+- **Freescout**: Help desk system with custom fields configured
+- **LiteLLM**: LLM proxy service accessible
+- **Baramundi**: Remote execution system (optional)
+
+### Test Environment Variables
+```bash
+export FREESCOUT_API_KEY="your-api-key"
+export LITELLM_API_KEY="your-api-key"
+export N8N_AUTH_TOKEN="your-auth-token"
+export POSTGRES_PASSWORD="your-password"
+```
+
+---
+
+## E2E Test Scenario: Complete Workflow
+
+### Test Case ID: E2E-001
+**Objective**: Full workflow from ticket creation through knowledge base update
+
+---
+
+## 1. Setup Phase
+
+### Pre-Test Validation
+```bash
+# Verify all services are running
+./tests/curl-test-collection.sh
+
+# Clear test data from previous runs (optional)
+docker-compose exec postgres psql -U kb_user -d n8n_kb -c \
+  "DELETE FROM knowledge_base_updates WHERE created_at > NOW() - INTERVAL '1 hour';"
+
+docker-compose exec postgres psql -U kb_user -d n8n_kb -c \
+  "DELETE FROM ticket_audit WHERE ticket_id LIKE 'TEST_%';"
+```
+
+### Test Data Preparation
+- Create test customer: `test@example.com` in Freescout (if not exists)
+- Verify custom fields exist in Freescout:
+  - `AI_SUGGESTION` (text field)
+  - `AI_SUGGESTION_STATUS` (enum: PENDING, APPROVED, REJECTED, EXECUTED)
+  - `AI_CONFIDENCE` (number field)
+
+---
+
+## 2. Workflow A: Ticket Analysis & AI Suggestion
+
+### Test Step 1: Create Test Ticket in Freescout
+
+**Action**: Create new ticket via Freescout UI or API
+```bash
+curl -X POST https://ekshelpdesk.fft-it.de/api/v1/mailboxes/1/conversations \
+  -H "Authorization: Bearer $FREESCOUT_API_KEY" \
+  -H "Content-Type: application/json" \
+  -d '{
+    "subject": "TEST_E2E_001: Drucker funktioniert nicht",
+    "body": "Jeder Druck-Befehl wird abgelehnt. Fehlercode 5. Bitte schnell lösen!",
+    "customer_email": "test@example.com",
+    "mailbox_id": 1
+  }'
+```
+
+**Expected Outcome**:
+- HTTP Status: 201 (Created)
+- Response contains `conversation_id`
+- Ticket visible in Freescout dashboard
+
+**Validation**:
+```bash
+# Get the created ticket ID from response
+TICKET_ID="<conversation_id_from_response>"
+echo "Created Ticket: $TICKET_ID"
+```
+
+---
+
+### Test Step 2: Wait for Workflow A Cycle (5 minutes)
+
+**Process Flow**:
+1. n8n Workflow A triggers every 5 minutes
+2. Fetches new tickets from Freescout
+3. Sends ticket text to LiteLLM for analysis
+4. Updates custom fields with AI suggestion
+
+**Wait Time**: 5 minutes (plus 1 min buffer = 6 minutes total)
+
+**During Wait - Monitor Logs**:
+```bash
+# Monitor n8n logs for workflow execution
+docker-compose logs -f n8n | grep -i "workflow_a\|ticket\|analysis"
+
+# Check PostgreSQL audit log
+docker-compose exec postgres psql -U kb_user -d n8n_kb -c \
+  "SELECT * FROM ticket_audit WHERE ticket_id = '$TICKET_ID' ORDER BY created_at DESC;"
+```
+
+**Expected n8n Logs**:
+- "Processing ticket: TEST_E2E_001"
+- "Calling LiteLLM API for analysis"
+- "Analysis complete: category=Hardware, confidence=0.92"
+- "Updated custom fields in Freescout"
+
+---
+
+### Test Step 3: Verify AI Suggestion in Freescout
+
+**Action**: Check ticket custom fields
+```bash
+curl -H "Authorization: Bearer $FREESCOUT_API_KEY" \
+  https://ekshelpdesk.fft-it.de/api/v1/conversations/$TICKET_ID
+```
+
+**Expected Response Fields**:
+```json
+{
+  "conversation": {
+    "id": "TICKET_ID",
+    "subject": "TEST_E2E_001: Drucker funktioniert nicht",
+    "custom_fields": {
+      "AI_SUGGESTION": "Hardware Problem: Drucker-Treiber fehlerhaft oder beschädigt. Empfohlene Lösung: 1) Drucker neustarten, 2) Treiber neu installieren",
+      "AI_SUGGESTION_STATUS": "PENDING",
+      "AI_CONFIDENCE": 0.92
+    }
+  }
+}
+```
+
+**Validation Checklist**:
+- ✅ `AI_SUGGESTION` is not empty
+- ✅ `AI_SUGGESTION_STATUS` = "PENDING"
+- ✅ `AI_CONFIDENCE` between 0.7 and 1.0
+- ✅ Suggestion text contains problem category and solution
+
+**Alternative: Manual Check in UI**:
+- Open ticket in Freescout
+- Scroll to custom fields section
+- Verify all three fields populated with expected values
+
+---
+
+## 3. Workflow B: Approval & Execution
+
+### Test Step 4: Approve AI Suggestion
+
+**Action**: Update custom field via API
+```bash
+curl -X PUT https://ekshelpdesk.fft-it.de/api/v1/conversations/$TICKET_ID \
+  -H "Authorization: Bearer $FREESCOUT_API_KEY" \
+  -H "Content-Type: application/json" \
+  -d '{
+    "custom_fields": {
+      "AI_SUGGESTION_STATUS": "APPROVED"
+    }
+  }'
+```
+
+**Expected Outcome**:
+- HTTP Status: 200 (OK)
+- Custom field updated in Freescout
+
+**Validation**:
+```bash
+# Verify field was updated
+curl -H "Authorization: Bearer $FREESCOUT_API_KEY" \
+  https://ekshelpdesk.fft-it.de/api/v1/conversations/$TICKET_ID | \
+  jq '.conversation.custom_fields.AI_SUGGESTION_STATUS'
+# Expected output: "APPROVED"
+```
+
+---
+
+### Test Step 5: Wait for Workflow B Cycle (2 minutes)
+
+**Process Flow**:
+1. n8n Workflow B triggers every 2 minutes
+2. Fetches tickets with `AI_SUGGESTION_STATUS = APPROVED`
+3. Executes action based on category:
+   - **Hardware/Remote**: Create Baramundi job
+   - **Software/Config**: Send support email
+   - **Knowledge**: Update KB directly
+4. Updates status to `EXECUTED`
+
+**Wait Time**: 2 minutes (plus 1 min buffer = 3 minutes total)
+
+**During Wait - Monitor Logs**:
+```bash
+# Check n8n workflow B execution
+docker-compose logs -f n8n | grep -i "workflow_b\|approved\|executing"
+
+# Check for Baramundi job creation (if applicable)
+curl https://baramundi-api.example.com/jobs \
+  -H "Authorization: Bearer $BARAMUNDI_TOKEN" | \
+  jq '.jobs[] | select(.ticket_id == "'"$TICKET_ID"'")'
+```
+
+---
+
+### Test Step 6: Verify Execution Status
+
+**Action**: Check ticket status
+```bash
+curl -H "Authorization: Bearer $FREESCOUT_API_KEY" \
+  https://ekshelpdesk.fft-it.de/api/v1/conversations/$TICKET_ID
+```
+
+**Expected Response**:
+```json
+{
+  "conversation": {
+    "custom_fields": {
+      "AI_SUGGESTION_STATUS": "EXECUTED",
+      "EXECUTION_TIMESTAMP": "2026-03-16T14:35:00Z"
+    }
+  }
+}
+```
+
+**Validation Checklist**:
+- ✅ `AI_SUGGESTION_STATUS` = "EXECUTED"
+- ✅ `EXECUTION_TIMESTAMP` is recent
+- ✅ No errors in n8n logs
+
+---
+
+## 4. Workflow C: Knowledge Base Update
+
+### Test Step 7: Wait for Workflow C Cycle (1 minute)
+
+**Process Flow**:
+1. n8n Workflow C triggers every 1 minute
+2. Fetches executed tickets
+3. Extracts: Problem, Solution, Category
+4. Inserts into PostgreSQL `knowledge_base_updates` table
+5. Triggers Milvus vector DB embedding and indexing
+
+**Wait Time**: 1 minute (plus 1 min buffer = 2 minutes total)
+
+---
+
+### Test Step 8: Verify Knowledge Base Entry
+
+**Action 1**: Check PostgreSQL insert
+```bash
+docker-compose exec postgres psql -U kb_user -d n8n_kb -c \
+  "SELECT id, ticket_id, problem, solution, category, frequency, created_at
+   FROM knowledge_base_updates
+   WHERE ticket_id = '$TICKET_ID'
+   ORDER BY created_at DESC LIMIT 1;"
+```
+
+**Expected Output**:
+```
+ id | ticket_id | problem | solution | category | frequency | created_at
+----+-----------+---------+----------+----------+-----------+---------------------
+ 42 | TEST...   | Drucker | Treiber  | Hardware | 1         | 2026-03-16 14:35:00
+```
+
+**Action 2**: Check Milvus vector DB
+```bash
+# Query Milvus for the new KB entry
+curl -X POST http://127.0.0.1:19530/v1/search \
+  -H "Content-Type: application/json" \
+  -d '{
+    "collection_name": "knowledge_base",
+    "vectors": [
+      "vector_embedding_of_problem_text"
+    ],
+    "top_k": 10,
+    "output_fields": ["id", "ticket_id", "problem", "solution", "similarity"]
+  }' | jq '.'
+```
+
+**Expected Response**:
+```json
+{
+  "results": [
+    {
+      "id": 42,
+      "ticket_id": "TEST_E2E_001",
+      "problem": "Drucker funktioniert nicht - Fehlercode 5",
+      "solution": "Drucker neustarten und Treiber neu installieren",
+      "similarity": 0.98
+    }
+  ]
+}
+```
+
+**Validation Checklist**:
+- ✅ Record exists in `knowledge_base_updates` table
+- ✅ Fields populated: problem, solution, category, frequency
+- ✅ Record appears in Milvus search results
+- ✅ Similarity score >= 0.95 for exact match
+
+---
+
+## 5. Vector DB Search Test
+
+### Test Step 9: Search Similar Problem
+
+**Objective**: Test that similar problems can be found via vector similarity
+
+**Action**: Query Milvus with similar but different text
+```bash
+# Prepare embedding for similar problem
+curl -X POST http://127.0.0.1:19530/v1/search \
+  -H "Content-Type: application/json" \
+  -d '{
+    "collection_name": "knowledge_base",
+    "vectors": [
+      "vector_for_query: Druckerprobleme beim Ausdrucken mit Fehler"
+    ],
+    "top_k": 5,
+    "output_fields": ["id", "ticket_id", "problem", "solution", "category"]
+  }'
+```
+
+**Expected Outcome**:
+- Top result is our test KB entry
+- Similarity score > 0.85
+- Same category returned (Hardware)
+
+**Expected Result**:
+```json
+{
+  "results": [
+    {
+      "rank": 1,
+      "similarity": 0.91,
+      "ticket_id": "TEST_E2E_001",
+      "problem": "Drucker funktioniert nicht - Fehlercode 5",
+      "solution": "Drucker neustarten und Treiber neu installieren",
+      "category": "Hardware"
+    },
+    {
+      "rank": 2,
+      "similarity": 0.78,
+      "ticket_id": "OTHER_TICKET",
+      "problem": "Netzwerkdrucker nicht erreichbar",
+      "solution": "IP-Konfiguration prüfen"
+    }
+  ]
+}
+```
+
+**Validation**:
+- ✅ Test entry ranks in top 3
+- ✅ Similarity > 0.85
+- ✅ Relevant results returned
+
+---
+
+## 6. Failure Scenarios
+
+### Scenario F1: Service Unavailable
+
+**Test**: What happens if LiteLLM is unreachable?
+
+**Action**: Stop LiteLLM service
+```bash
+docker-compose stop litellm
+```
+
+**Expected Behavior**:
+- n8n Workflow A fails gracefully
+- Error logged in PostgreSQL error_log table
+- Ticket status remains PENDING
+- After service recovery, retry happens automatically
+
+**Verify**:
+```bash
+docker-compose exec postgres psql -U kb_user -d n8n_kb -c \
+  "SELECT error_message, retry_count FROM error_log WHERE ticket_id = '$TICKET_ID';"
+```
+
+**Cleanup**: Restart service
+```bash
+docker-compose up -d litellm
+```
+
+---
+
+### Scenario F2: Invalid Custom Field
+
+**Test**: What if custom field value is invalid?
+
+**Action**: Set invalid status value
+```bash
+curl -X PUT https://ekshelpdesk.fft-it.de/api/v1/conversations/$TICKET_ID \
+  -H "Authorization: Bearer $FREESCOUT_API_KEY" \
+  -d '{
+    "custom_fields": {
+      "AI_SUGGESTION_STATUS": "INVALID_VALUE"
+    }
+  }'
+```
+
+**Expected Behavior**:
+- Workflow B ignores ticket
+- Error logged
+- Manual intervention required
+
+---
+
+## 7. Test Cleanup
+
+### Post-Test Actions
+```bash
+# Archive test data
+docker-compose exec postgres psql -U kb_user -d n8n_kb -c \
+  "UPDATE ticket_audit
+   SET archived = true
+   WHERE ticket_id LIKE 'TEST_%';"
+
+# Optional: Delete test ticket from Freescout
+curl -X DELETE https://ekshelpdesk.fft-it.de/api/v1/conversations/$TICKET_ID \
+  -H "Authorization: Bearer $FREESCOUT_API_KEY"
+
+# Verify cleanup
+docker-compose exec postgres psql -U kb_user -d n8n_kb -c \
+  "SELECT COUNT(*) as active_test_tickets FROM ticket_audit
+   WHERE ticket_id LIKE 'TEST_%' AND archived = false;"
+```
+
+---
+
+## 8. Test Metrics & Success Criteria
+
+### Timing Metrics
+| Phase | Expected Duration | Tolerance |
+|-------|------------------|-----------|
+| Setup | - | - |
+| Workflow A (Analysis) | 5 min | ±1 min |
+| Workflow B (Execution) | 2 min | ±30 sec |
+| Workflow C (KB Update) | 1 min | ±30 sec |
+| **Total End-to-End** | **~8 min** | **±2 min** |
+
+### Success Criteria
+- ✅ All services respond to health checks
+- ✅ Ticket created successfully in Freescout
+- ✅ AI analysis completes within 5 min
+- ✅ Confidence score >= 0.7
+- ✅ Approval triggers execution within 2 min
+- ✅ KB entry created in PostgreSQL
+- ✅ KB entry indexed in Milvus
+- ✅ Vector search returns relevant results (similarity > 0.85)
+- ✅ All audit logs recorded correctly
+- ✅ No unhandled errors in logs
+
+### Performance Targets
+- API response time: < 2 seconds
+- LiteLLM inference: < 10 seconds
+- Milvus embedding + indexing: < 5 seconds
+- PostgreSQL inserts: < 1 second
+
+---
+
+## 9. Test Execution Checklist
+
+```bash
+# Start here:
+[ ] Verify all services running: ./tests/curl-test-collection.sh
+[ ] Create test ticket (capture TICKET_ID)
+[ ] Wait 6 minutes for Workflow A
+[ ] Verify AI_SUGGESTION populated
+[ ] Approve ticket (update AI_SUGGESTION_STATUS)
+[ ] Wait 3 minutes for Workflow B
+[ ] Verify AI_SUGGESTION_STATUS = EXECUTED
+[ ] Wait 2 minutes for Workflow C
+[ ] Verify PostgreSQL kb entry exists
+[ ] Verify Milvus vector search works
+[ ] Review all audit logs
+[ ] Clean up test data
+[ ] Document any issues found
+```
+
+---
+
+## 10. Troubleshooting Guide
+
+### Issue: AI_SUGGESTION not populated after 5 min
+
+**Check**:
+1. n8n logs: `docker-compose logs n8n | grep -i error`
+2. Freescout API connectivity: `curl https://ekshelpdesk.fft-it.de/healthz`
+3. LiteLLM service: `curl http://llm.eks-ai.apps.asgard.eks-lnx.fft-it.de/health`
+4. Custom field exists: Check Freescout admin panel
+
+### Issue: Workflow B not executing
+
+**Check**:
+1. Ticket status field correct: `curl -H "Auth: Bearer $KEY" https://ekshelpdesk.fft-it.de/api/v1/conversations/$TICKET_ID | jq '.custom_fields.AI_SUGGESTION_STATUS'`
+2. n8n Workflow B enabled: Check n8n UI
+3. Error logs: `docker-compose exec postgres psql -U kb_user -d n8n_kb -c "SELECT * FROM error_log ORDER BY created_at DESC LIMIT 5;"`
+
+### Issue: Milvus search returns no results
+
+**Check**:
+1. Milvus running: `curl http://127.0.0.1:19530/v1/healthz`
+2. KB collection exists: See Milvus documentation for collection listing
+3. Vector embedding generated: Check PostgreSQL `knowledge_base_updates.vector_embedding`
+
+---
+
+## Appendix A: API Endpoints Reference
+
+| Service | Endpoint | Purpose |
+|---------|----------|---------|
+| Freescout | `https://ekshelpdesk.fft-it.de/api/v1/conversations` | Ticket management |
+| LiteLLM | `http://llm.eks-ai.apps.asgard.eks-lnx.fft-it.de/v1/chat/completions` | AI analysis |
+| Milvus | `http://127.0.0.1:19530/v1/search` | Vector DB search |
+| PostgreSQL | `localhost:5432` | Data persistence |
+| n8n | `http://localhost:5678` | Workflow orchestration |
+
+---
+
+**Document Version**: 1.0
+**Last Updated**: 2026-03-16
+**Author**: QA/Testing Agent
+**Status**: Ready for Testing