Incident Management
Incident Severity
| Level | Impact | Response Time |
|---|
| SEV1 | Complete outage | Immediate |
| SEV2 | Major degradation | < 15 min |
| SEV3 | Minor degradation | < 1 hour |
| SEV4 | Low impact | Next business day |
Incident Response
1. Detect
- Monitoring alerts
- Customer reports
- Error logs
2. Triage
- Assess severity
- Assign incident commander
- Create communication channel
3. Investigate
- Check recent changes
- Review logs and metrics
- Identify root cause
4. Mitigate
- Apply quick fix
- Rollback if needed
- Communicate status
5. Resolve
- Confirm fix
- Monitor for recurrence
- Close incident
6. Learn
- Post-mortem meeting
- Document findings
- Create action items
Post-Mortem Template
# Post-Mortem: [Incident Title]
## Summary
[Brief description of what happened]
## Timeline
- HH:MM - [Event]
- HH:MM - [Event]
- HH:MM - [Resolution]
## Impact
- Duration: [X hours]
- Users affected: [X]
- Revenue impact: [if applicable]
## Root Cause
[What caused this incident]
## Contributing Factors
- [Factor 1]
- [Factor 2]
## What Went Well
- [Positive 1]
- [Positive 2]
## What Could Be Improved
- [Improvement 1]
- [Improvement 2]
## Action Items
- [ ] [Action 1] - Owner: [Name]
- [ ] [Action 2] - Owner: [Name]
Blameless Culture
- Focus on systems, not people
- "What failed?" not "Who failed?"
- Share learnings openly
- Celebrate near-misses
