Incident Response

---
name: incident-response
description: Triage a production incident without making things worse. Use when production is down, errors are spiking, or users report an outage and the user needs structured help.
---

# Incident Response

Restore service first, understand second, perfect never. No code edits during triage.

## Phase 1 — Assess (minutes)

1. What is the user impact: who, what %, since when? Which endpoints/features?
2. Check the obvious in parallel:
   - Recent deploys or config changes: `git log --since="24 hours ago" --oneline`, CI deploy history
   - Infra health: `kubectl get pods`, `docker ps`, dashboard status pages of critical dependencies
   - Error rate and type in logs/metrics: 5xx spike? Timeouts? One error message dominating?
3. Classify: deploy regression / dependency down / resource exhaustion / data issue / traffic spike.

## Phase 2 — Mitigate

Prefer reversible actions, cheapest first:
- **Rollback** the suspect deploy — this fixes most deploy regressions in minutes.
- **Restart/scale** if resource exhaustion (OOM kills, connection pool full).
- **Feature-flag off** the suspect change if flags exist.
- Confirm mitigation worked by watching the error rate, not by hoping.

## Phase 3 — Diagnose (service restored or stable)

1. Reproduce in staging or against the reverted version.
2. Follow the systematic-debugging skill: hypotheses, one experiment at a time.
3. Identify the root cause with evidence from logs/traces, not plausibility.

## Phase 4 — Follow-through

- Permanent fix with a regression test.
- Timeline: detected → mitigated → root-caused → fixed, with timestamps.
- Action items: what monitoring/alert/test would have caught this sooner?

## Never during an incident

- Deploy an untested "quick fix" to production under pressure
- Make multiple simultaneous changes (you will not know which helped)
- Delete logs or restart evidence away before capturing it