515 lines
15 KiB
Markdown
515 lines
15 KiB
Markdown
# Go-Live Checklist
|
|
|
|
## Overview
|
|
|
|
This checklist ensures a smooth transition from staging to production. Follow all phases sequentially: Pre-Launch, Go-Live Day, Launch Period, and Post-Launch Monitoring.
|
|
|
|
---
|
|
|
|
## Phase 1: One Week Before Go-Live
|
|
|
|
Timeline: T-7 days
|
|
|
|
### Pre-Deployment Verification
|
|
|
|
- [ ] **E2E Tests Passed (100%)**
|
|
- All workflow tests successful: `bash tests/curl-test-collection.sh`
|
|
- No critical bugs or failures
|
|
- Test results documented
|
|
|
|
- [ ] **Staging Environment Verified**
|
|
- Deploy to staging identical to production
|
|
- Run full load test (simulate 100+ concurrent tickets)
|
|
- Verify integration with TEST Freescout account
|
|
- Verify integration with TEST Baramundi account
|
|
- All workflows processing correctly in staging
|
|
|
|
- [ ] **Production Database Setup**
|
|
- PostgreSQL audit schema initialized
|
|
- Backup strategy configured and tested
|
|
- Database performance baseline recorded
|
|
- Disk space verified (minimum 20GB available)
|
|
|
|
- [ ] **API Credentials Verified**
|
|
- Freescout API key tested and active
|
|
- Freescout custom fields created and working
|
|
- Baramundi API key tested and active
|
|
- n8n encryption key generated and secure
|
|
|
|
### Team Readiness
|
|
|
|
- [ ] **Team Training Completed**
|
|
- Operations team trained on:
|
|
- System architecture overview
|
|
- Deployment and rollback procedures
|
|
- Monitoring dashboard usage
|
|
- Alert response procedures
|
|
- Escalation paths
|
|
- Support team trained on:
|
|
- Workflow functionality overview
|
|
- Expected behavior and timing
|
|
- How to verify system health
|
|
- When to escalate issues
|
|
|
|
- [ ] **Documentation Review**
|
|
- All team members reviewed DEPLOYMENT.md
|
|
- All team members reviewed MONITORING.md
|
|
- Runbooks reviewed and acknowledged
|
|
- Contact list updated (on-call schedule)
|
|
|
|
- [ ] **Backup Strategy Finalized**
|
|
- Daily backup schedule defined
|
|
- Backup retention policy set (7 days minimum)
|
|
- Backup restore procedure tested
|
|
- Backup storage verified (separate location from production)
|
|
|
|
### Risk Mitigation
|
|
|
|
- [ ] **Rollback Plan Confirmed**
|
|
- Rollback procedures documented
|
|
- Rollback tested in staging environment
|
|
- Estimated rollback time: < 30 minutes
|
|
- All team members trained on rollback
|
|
|
|
- [ ] **Communication Plan Ready**
|
|
- Stakeholder notification list prepared
|
|
- Status page update process defined
|
|
- Internal update frequency established (30min intervals initially)
|
|
- Escalation contacts verified
|
|
|
|
- [ ] **Monitoring & Alerting**
|
|
- All monitoring dashboards configured
|
|
- Alert recipients confirmed
|
|
- Alert thresholds set and validated
|
|
- On-call rotation established
|
|
|
|
---
|
|
|
|
## Phase 2: Go-Live Day
|
|
|
|
Timeline: T-0 (Launch day)
|
|
|
|
### Pre-Launch Checks (T-2 hours)
|
|
|
|
- [ ] **Final System Status**
|
|
- All Docker services running and healthy
|
|
- `docker-compose ps` output verified
|
|
- All services show "Up (healthy)"
|
|
- No services in "Restarting" state
|
|
|
|
- [ ] **Service Health Verification**
|
|
- n8n health check: `curl http://localhost:5678/api/v1/health`
|
|
- PostgreSQL connection: `docker-compose exec postgres pg_isready`
|
|
- Milvus connectivity: Vector DB responding
|
|
- External integrations reachable (Freescout, Baramundi)
|
|
|
|
- [ ] **Database Integrity**
|
|
- Audit schema verified: `SELECT COUNT(*) FROM audit.workflows;`
|
|
- No corruption or errors in logs
|
|
- Backup created and verified: `ls -lh backups/`
|
|
- Backup restore tested
|
|
|
|
- [ ] **n8n Workflows Status**
|
|
- All 3 workflows imported successfully
|
|
- Workflow A (Mail Processing): Ready
|
|
- Workflow B (Approval Execution): Ready
|
|
- Workflow C (KB Update): Ready
|
|
- All workflows set to Inactive (will activate after final check)
|
|
|
|
- [ ] **Monitoring System Active**
|
|
- Monitoring dashboard accessible
|
|
- All metric collectors running
|
|
- Alert system armed and tested
|
|
- Log aggregation working (docker-compose logs verified)
|
|
|
|
- [ ] **Final Pre-Launch Meeting**
|
|
- All team members present and ready
|
|
- Roles and responsibilities confirmed:
|
|
- Platform Lead: Overall coordination
|
|
- n8n Administrator: Workflow management
|
|
- Database Administrator: Database monitoring
|
|
- System Administrator: Infrastructure monitoring
|
|
- Support Lead: User support readiness
|
|
- Communication channels verified (Slack, phone, etc.)
|
|
|
|
### Launch Window (T-0 hours)
|
|
|
|
- [ ] **Final Backup (T-15 minutes)**
|
|
- Backup created immediately before activation
|
|
- Backup file verified and tested
|
|
- Backup location: `backups/pre-golive-backup-$(date +%Y%m%d-%H%M%S).sql`
|
|
|
|
- [ ] **Activate Workflows (T-0 minutes)**
|
|
- n8n Dashboard accessed
|
|
- Workflow A (Mail Processing) activated:
|
|
- Toggle "Active" switch ON
|
|
- Verify activation confirmed in UI
|
|
- Check logs: `docker-compose logs -f n8n | grep "Workflow A"`
|
|
- Workflow B (Approval Execution) activated
|
|
- Workflow C (KB Update) activated
|
|
- All three workflows showing "Active" status
|
|
|
|
- [ ] **Launch Announcement**
|
|
- Internal team notified: "System is LIVE"
|
|
- Stakeholders notified of go-live
|
|
- Status page updated: "System operational"
|
|
- Time of launch recorded: __________
|
|
|
|
- [ ] **Confirm System Accepting Requests**
|
|
- Send test email to Freescout inbox
|
|
- Verify ticket created in Freescout
|
|
- Verify n8n workflow triggered (check logs)
|
|
- Verify workflow execution started
|
|
|
|
---
|
|
|
|
## Phase 3: Launch Period Monitoring (First 24 Hours)
|
|
|
|
Timeline: T+0 to T+24 hours
|
|
|
|
### Continuous Monitoring (Every 15 minutes)
|
|
|
|
- [ ] **n8n Workflow Execution**
|
|
- Command: `docker-compose logs -f n8n | tail -50`
|
|
- Check for:
|
|
- No error messages
|
|
- Workflows executing successfully
|
|
- No hung or stuck executions
|
|
- Log location: `/d/n8n-compose/logs/n8n.log`
|
|
|
|
- [ ] **Freescout Integration**
|
|
- New tickets arriving in system
|
|
- Custom fields populated correctly
|
|
- No integration errors in Freescout logs
|
|
- Ticket processing speed acceptable
|
|
|
|
- [ ] **Baramundi Job Queue**
|
|
- Check job queue status
|
|
- Verify jobs accepted from n8n
|
|
- Monitor job completion rate
|
|
- Check for failed jobs
|
|
|
|
- [ ] **Alert System**
|
|
- All critical alerts functioning
|
|
- No false positive alerts
|
|
- Escalation procedures working
|
|
- On-call team responsive
|
|
|
|
- [ ] **Database Performance**
|
|
- Query performance acceptable
|
|
- No locks or deadlocks
|
|
- Disk space usage normal
|
|
- Command: `docker-compose exec postgres pg_stat_statements`
|
|
|
|
### Hourly System Status Report (First 6 hours)
|
|
|
|
Document every hour:
|
|
|
|
**Hour 1 (T+1h)**
|
|
- Total tickets processed: _____
|
|
- Total workflows executed: _____
|
|
- Failed executions: _____
|
|
- System health: [ ] Green [ ] Yellow [ ] Red
|
|
- Issues encountered: _____
|
|
|
|
**Hour 2 (T+2h)**
|
|
- Total tickets processed: _____
|
|
- Total workflows executed: _____
|
|
- Failed executions: _____
|
|
- System health: [ ] Green [ ] Yellow [ ] Red
|
|
- Issues encountered: _____
|
|
|
|
**Hour 3-6 (T+3h to T+6h)**
|
|
- Repeat above for each hour
|
|
- Escalate any issues immediately
|
|
- Document all changes or interventions
|
|
|
|
### Functional Validation (T+2 hours and T+12 hours)
|
|
|
|
**After 2 hours:**
|
|
- [ ] **AI Suggestions Displayed**
|
|
- Sample processed tickets show AI suggestions
|
|
- Suggestion accuracy acceptable
|
|
- Custom field updated with ai_suggestion
|
|
- Performance acceptable (< 5 second processing time)
|
|
|
|
- [ ] **Approval Workflow Operating**
|
|
- HIGH priority tickets flagged for approval
|
|
- Approval custom field populated
|
|
- Notifications sent to approvers
|
|
- Approvals received and reflected in system
|
|
|
|
- [ ] **Knowledge Base Updates**
|
|
- KB articles being created/updated
|
|
- Vector embeddings generated (Milvus)
|
|
- PostgreSQL KB table growing
|
|
- Query: `SELECT COUNT(*) FROM audit.kb_articles;`
|
|
|
|
**After 12 hours (overnight validation):**
|
|
- [ ] **Validate Overnight Processing**
|
|
- All workflows executed correctly overnight
|
|
- No race conditions or deadlocks occurred
|
|
- Database backups completed successfully
|
|
- All alerts functioned as expected
|
|
|
|
### Critical Metrics (Monitor Continuously)
|
|
|
|
```bash
|
|
# Check n8n workflow execution rate
|
|
curl -H "X-N8N-API-KEY: $N8N_API_KEY" \
|
|
http://localhost:5678/api/v1/executions?limit=100 | jq '.executions | length'
|
|
|
|
# Check database growth
|
|
docker exec n8n-postgres psql -U n8n_user -d n8n_production -c \
|
|
"SELECT schemaname, tablename, pg_size_pretty(pg_total_relation_size(schemaname||'.'||tablename))
|
|
FROM pg_tables ORDER BY pg_total_relation_size(schemaname||'.'||tablename) DESC LIMIT 10;"
|
|
|
|
# Monitor CPU/Memory
|
|
docker stats --no-stream
|
|
```
|
|
|
|
### Incident Response (If Issues Occur)
|
|
|
|
**Critical Issue (System Down):**
|
|
1. Immediately notify team lead
|
|
2. Assess severity and scope
|
|
3. Execute rollback if necessary (see DEPLOYMENT.md)
|
|
4. Document incident details
|
|
5. Begin root cause analysis
|
|
|
|
**Performance Degradation:**
|
|
1. Check system resources: `docker stats`
|
|
2. Check database locks: `docker-compose logs postgres | grep LOCK`
|
|
3. Scale resources if needed: `docker-compose up -d --scale n8n=2`
|
|
4. Monitor improvement
|
|
|
|
**Integration Failures:**
|
|
1. Verify API credentials still valid
|
|
2. Check external service status
|
|
3. Review integration logs
|
|
4. Test connectivity manually
|
|
5. Retry or escalate
|
|
|
|
---
|
|
|
|
## Phase 4: Post-24 Hour Validation (T+24 to T+7 days)
|
|
|
|
### Day 2 Validation (T+24 hours)
|
|
|
|
- [ ] **Verify KI Suggestions Working**
|
|
- Sample 10 random processed tickets
|
|
- AI suggestions present and relevant
|
|
- Suggestion accuracy rate > 80%
|
|
- Processing time < 5 seconds average
|
|
- Document findings: ________
|
|
|
|
- [ ] **Approval Workflow Performance**
|
|
- [ ] All HIGH priority tickets flagged
|
|
- [ ] Approval response time < 2 hours
|
|
- [ ] Approval completion rate > 95%
|
|
- [ ] No pending approvals > 4 hours old
|
|
- Total approvals processed: _____
|
|
- Approval success rate: _____%
|
|
|
|
- [ ] **Baramundi Integration Validation**
|
|
- [ ] Jobs submitted successfully
|
|
- [ ] Job queue processing normally
|
|
- [ ] Job completion rate > 90%
|
|
- [ ] No stuck or failed jobs
|
|
- Total jobs processed: _____
|
|
- Job success rate: _____%
|
|
|
|
- [ ] **Knowledge Base Growth**
|
|
- [ ] KB articles being created
|
|
- [ ] Vector embeddings calculated
|
|
- [ ] Query performance acceptable
|
|
- Total KB articles: _____
|
|
- Total embeddings: _____
|
|
- Query response time: _____ ms
|
|
|
|
- [ ] **System Stability**
|
|
- [ ] No service crashes
|
|
- [ ] No memory leaks
|
|
- [ ] Disk usage normal
|
|
- [ ] Database integrity verified
|
|
- [ ] No orphaned records
|
|
|
|
### Day 7 Comprehensive Review (T+7 days)
|
|
|
|
- [ ] **Collect Statistics**
|
|
|
|
**Email Processing:**
|
|
- Total emails processed: _____
|
|
- Success rate: _____%
|
|
- Average processing time: _____ seconds
|
|
- Error rate: _____%
|
|
|
|
**AI Suggestions:**
|
|
- Total suggestions generated: _____
|
|
- Acceptance rate: _____%
|
|
- Average accuracy: _____%
|
|
- Processing time p95: _____ seconds
|
|
|
|
**Approvals:**
|
|
- Total approval requests: _____
|
|
- Total approvals completed: _____
|
|
- Approval completion rate: _____%
|
|
- Average response time: _____ minutes
|
|
- HIGH priority count: _____
|
|
|
|
**Baramundi Jobs:**
|
|
- Total jobs submitted: _____
|
|
- Total jobs completed: _____
|
|
- Success rate: _____%
|
|
- Failed jobs: _____
|
|
|
|
**Knowledge Base:**
|
|
- Total KB articles created: _____
|
|
- Total articles updated: _____
|
|
- Total searches: _____
|
|
- Average search response: _____ ms
|
|
|
|
- [ ] **Performance Analysis**
|
|
- [ ] n8n CPU usage normal: _____ %
|
|
- [ ] n8n Memory usage normal: _____ MB
|
|
- [ ] PostgreSQL query time p95: _____ ms
|
|
- [ ] Database size: _____ GB
|
|
- [ ] Backup size: _____ GB
|
|
|
|
- [ ] **Team Feedback Collected**
|
|
- [ ] Operations team feedback: ________
|
|
- [ ] Support team feedback: ________
|
|
- [ ] End user feedback: ________
|
|
- [ ] Issues encountered: ________
|
|
- [ ] Improvement suggestions: ________
|
|
|
|
- [ ] **Issue Resolution Status**
|
|
- [ ] All critical issues resolved
|
|
- [ ] All high priority issues resolved
|
|
- [ ] Medium priority issues tracked
|
|
- [ ] Minor issues documented for next release
|
|
- Issue tracking document: __________
|
|
|
|
### Go-Live Success Criteria - Final Sign-Off
|
|
|
|
All criteria must be met to declare go-live successful:
|
|
|
|
- [ ] **Stability (99% uptime minimum)**
|
|
- System remained operational for 7 consecutive days
|
|
- Unplanned downtime < 14.4 minutes total
|
|
- All services restarted cleanly without issues
|
|
|
|
- [ ] **Functionality (100% requirements met)**
|
|
- Mail processing working correctly
|
|
- AI suggestions functional and accurate
|
|
- Approval workflow operational
|
|
- Baramundi job submission successful
|
|
- KB updates functioning
|
|
|
|
- [ ] **Performance (Acceptable for workload)**
|
|
- Average email processing < 5 seconds
|
|
- Average workflow execution < 10 seconds
|
|
- Database queries < 1 second (p95)
|
|
- No performance degradation observed
|
|
|
|
- [ ] **Data Integrity (100% accuracy)**
|
|
- All processed tickets correctly handled
|
|
- No duplicate records
|
|
- No data loss or corruption
|
|
- Audit trail complete and accurate
|
|
|
|
- [ ] **Monitoring (All systems active)**
|
|
- Real-time dashboards operational
|
|
- Alerts functioning correctly
|
|
- Logs aggregated and searchable
|
|
- Performance metrics recorded
|
|
|
|
- [ ] **Team Readiness (100% trained)**
|
|
- Operations team fully trained
|
|
- Support team fully trained
|
|
- All runbooks completed
|
|
- On-call schedule established
|
|
|
|
**Sign-Off By:**
|
|
|
|
Project Manager: _________________ Date: _______
|
|
|
|
Operations Lead: _________________ Date: _______
|
|
|
|
Technical Lead: _________________ Date: _______
|
|
|
|
---
|
|
|
|
## Ongoing Monitoring (Post Go-Live)
|
|
|
|
### Daily Checks (First 30 Days)
|
|
|
|
- [ ] Review system health dashboard
|
|
- [ ] Check backup completion status
|
|
- [ ] Review error logs for new issues
|
|
- [ ] Verify workflow execution metrics
|
|
- [ ] Check database growth rate
|
|
- [ ] Monitor alert frequency and relevance
|
|
|
|
### Weekly Checks (Ongoing)
|
|
|
|
- [ ] Generate performance report
|
|
- [ ] Review all system logs
|
|
- [ ] Verify backup restore capability
|
|
- [ ] Update documentation as needed
|
|
- [ ] Team retrospective meeting
|
|
- [ ] Plan for optimization improvements
|
|
|
|
### Monthly Reviews (Ongoing)
|
|
|
|
- [ ] Comprehensive system audit
|
|
- [ ] Capacity planning review
|
|
- [ ] Security assessment
|
|
- [ ] Performance optimization review
|
|
- [ ] Team training refresher (as needed)
|
|
- [ ] Update escalation procedures
|
|
|
|
---
|
|
|
|
## Contacts and Escalation
|
|
|
|
### Primary Contacts
|
|
|
|
**Project Manager:**
|
|
- Name: _____________________
|
|
- Phone: _____________________
|
|
- Email: _____________________
|
|
|
|
**Technical Lead:**
|
|
- Name: _____________________
|
|
- Phone: _____________________
|
|
- Email: _____________________
|
|
|
|
**On-Call Engineer:**
|
|
- Name: _____________________
|
|
- Phone: _____________________
|
|
- Email: _____________________
|
|
|
|
### Escalation Matrix
|
|
|
|
**Level 1 - Application Issue:**
|
|
- On-call engineer
|
|
- Response time: 15 minutes
|
|
|
|
**Level 2 - System Down:**
|
|
- Technical lead + On-call engineer
|
|
- Response time: 5 minutes
|
|
|
|
**Level 3 - Critical Data Loss:**
|
|
- Technical lead + Project manager + Database admin
|
|
- Response time: Immediate
|
|
|
|
---
|
|
|
|
## Related Documentation
|
|
|
|
- [DEPLOYMENT.md](DEPLOYMENT.md) - Deployment procedures and rollback
|
|
- [MONITORING.md](MONITORING.md) - Monitoring dashboard and alerts
|
|
- [ARCHITECTURE.md](ARCHITECTURE.md) - System architecture details
|
|
- [TROUBLESHOOTING.md](TROUBLESHOOTING.md) - Common issues and solutions
|