Download the template: Incident Response Runbook Template (PDF)
The costliest fifteen minutes of an outage or security alert rarely go to solving a hard engineering puzzle. More often, they disappear while people debate whether the problem counts as a major incident, who has the authority to roll back a release, or whether customers should hear anything yet. By the time someone steps up to coordinate, half an hour is already gone.
A working incident response runbook settles those coordination rules ahead of time, and keeps them short enough that an engineer paged out of sleep at 03:00 UTC can read one page and get moving. Alongside the PDF template above, you can generate service-specific procedures in our Incident Response Playbook Generator and pair them with our major incident communication templates.
Splitting Responder Roles Right Away
Even on a three-person engineering team, debugging database locks in a terminal while answering executive messages in Slack is how mistakes happen. Naming roles explicitly in the incident channel—even if the first responder holds two hats for ten minutes while backup wakes up—keeps troubleshooting and communication from stepping on each other.
One person acts as Incident Commander (IC), owning the severity call, high-level mitigation choices (rolling back versus pushing a forward fix, promoting a replica, or flipping a feature-flag kill switch), and clean shift handoffs. Good ICs stay out of the code and out of terminal windows; once the commander starts chasing stack traces, nobody is watching the clock. Hands-on diagnostics, containment, and service-specific operational runbooks belong to the Technical Lead across Ops or SecOps. Meanwhile, a Communications Lead publishes timed internal and external updates once the IC verifies the facts, and a Scribe pins UTC timestamps, hypotheses tested, commands run, and follow-up items in the channel so nobody has to piece the timeline together from memory two days later.
Picking One Severity Scale and Sticking to It
Some teams number incidents SEV1 through SEV4, while others (including our interactive playbook and communication generators) use zero-indexed S0 / SEV0 labels at the top. Either convention works fine; mixing two scales across paging rules, status pages, and postmortems does not. Pick one house scale, and whenever scope is still uncertain in the first few minutes, classify one tier higher until you can prove the blast radius is smaller.
| Severity Tier | Availability & Security Trigger Examples | Ack Target | Status Update Cadence |
|---|---|---|---|
| SEV1 / S0 (Critical) | Core login, checkout, or API down; active intrusion, confirmed data exfiltration, or multi-tenant data risk | ≤ 5 min (24x7 page) | Every 10-15 min until mitigated |
| SEV2 / S1 (Major) | Major feature broken or severe latency with no clean workaround; suspicious privileged auth or uncontained malware alert | ≤ 15 min (24x7 page) | Every 20-30 min |
| SEV3 / S2 (Moderate) | Partial degradation affecting a subset of users with a documented workaround; single-workstation EDR quarantine | ≤ 1 business hour | Every 30-60 min in channel |
| SEV4 / S3 (Minor) | Cosmetic bug, non-urgent background job retry, or informational security alert requiring routine triage | Next business day | Every 60-120 min or at ticket closure |
Where Availability Triage and Security Response Diverge
Habits built around pure uptime can hurt you badly during a security incident. When a container starts misbehaving, what is an operations engineer's first reflex? Usually to redeploy a clean image or reboot the host. For a memory leak, that solves the problem in two minutes. During an active intrusion, terminating the instance wipes volatile memory, open network sockets, and unshipped local logs before anyone knows how the intruder got in or what else they reached.
That is why your runbook needs an early fork right after initial triage. When you are dealing with an availability or reliability failure, getting customer traffic healthy comes first. Check whether a deployment, feature flag, configuration push, or upstream dependency changed inside the past two hours; if so, run your deployment backout plan, shed non-critical batch load, or fail over following your DR runbook while keeping application logs intact for daytime root-cause work.
When there is any sign of unauthorized access or account takeover, containment has to preserve evidence before cleaning anything up. Network-isolate affected hosts through EDR rather than terminating instances, and snapshot EBS or disk volumes alongside identity audit logs before deleting attacker inbox rules or OAuth grants (our compromised Microsoft 365 account checklist walks through that order step by step). Move responder chat and voice onto an out-of-band bridge if corporate Slack or email trust is in question, and bring in legal counsel and cyber-insurance breach hotlines before wiping affected machines.
Closing the Loop With Post-Incident Reviews
Within 48 business hours of resolving any critical or major incident, hold a 45-minute review aimed at systemic gaps rather than individual blame. Rebuild the UTC timeline from the Scribe's notes, check how much of your SLO error budget went to detection lag versus mitigation time, and turn what you learn into owned engineering tickets with concrete due dates. If you want an experienced team to help set up monitoring, practical on-call playbooks, and steady production support, take a look at briskData's application production support and security readiness services.