Good disaster recovery is mostly paperwork until the day you need it. A clear runbook turns a bad day into a sequence of small, reversible steps: declare, contain, fail over, verify, and return to steady state. This guide ships a template you can use as‑is, a filled example for a primary database outage, and a short testing cadence so the document stays true months from now.
What a DR runbook is (and is not)
A runbook is the agreed path the team follows under pressure. It names owners, flags, commands, and checks. It does not try to predict every cause or replace engineering judgment. Disaster classes vary—region loss, storage corruption, runaway costs—but the first ten minutes almost always rhyme.
Disaster recovery runbook: sections and owners
Identification
- Service name – exactly as operators know it
- Primary owner – on‑call rotation or team
- Criticality – user impact and RTO/RPO
Triggers
- Which alerts declare this runbook active
- Manual triggers (executive call, legal, safety)
Communications
- Where incidents live (channel, bridge, ticket)
- Stakeholder updates: who, when, and format
Mitigation steps
- Immediate safety checks (kill switches, flags)
- Containment actions to limit blast radius
- Failover / restore procedures with commands
Recovery & verification
- Back to steady state: how we know
- Data integrity checks and reconciling backlogs
- Roll back temporary mitigations
Aftercare
- Follow‑ups, owners, and due dates
- What to improve in this runbook
Template sections with field‑by‑field guidance
Use the sections above to document exactly how this service fails over and recovers. Copy the headings into your docs and fill them with the concrete names your team uses: on‑call rotation, feature flags, queue names, dashboards, and verification commands. If something is missing or unclear, that is a useful finding—add the flag or write the command now while the system is calm.
Filled example: recovering a primary DB in a region outage
This example shows the minimum useful level of detail. It does not depend on one vendor or a single product; adapt the names and commands to match yours.
Recovering the primary database during a region outage
- Declare the incident. Page database on‑call. Open
#inc‑db‑outageand a bridge. - Contain write traffic. Flip
WRITE_READS_ONLYflag to route writes to a queue; protect idempotency keys. - Promote replica. In the standby region, promote
db‑replica‑1to primary and record the new endpoint. - Redirect apps. Update the secrets/connection string and rotate app instances to pick up the change.
- Warm critical caches. Preload product, pricing, and account caches to reduce cold‑start pressure.
- Verify integrity. Run consistency checks; reconcile the write queue into the new primary.
- Lift mitigations. Remove
WRITE_READS_ONLYand close the incident once SLOs hold for one hour.
Your environment will differ. The point is to script the critical moves, name the flags, and make the safest path the obvious one under pressure.
Testing the runbook: tabletop and time‑boxed drills
Runbooks rot when they are never rehearsed. Small, frequent exercises beat a once‑a‑year “full DR test.” Put three on a calendar and protect the time: a quarterly tabletop to surface owners and decisions, a monthly DB failover in staging with a hard stop, and a weekly backup restore to prove RPO.
Keep the runbook current: drills and owners
| Cadence | Exercise | Owner | Success |
|---|---|---|---|
| Quarterly | Tabletop a cloud region loss | Infra on‑call lead | Roles, flags, and comms surfaced in 15 min |
| Monthly | Time‑boxed DB failover in staging | DBA / app owner | Service healthy on new primary in < 20 min |
| Weekly | Restore a named backup | Ops | Checksum verified; RPO within target |
When drills slip, the runbook rots. Make the calendar invite now and record who owns each check. Small, repeatable exercises beat an annual big‑bang test.
Keeping it current: owners, cadence, and change hooks
Assign an explicit owner for this document and connect it to change. When a team adds a dependency, ships a new feature flag, or rotates a secret, the runbook should change in the same pull request. A one‑line checklist in the PR template—“does this affect DR?”—keeps the habit alive.
Downloads
You can copy this template into your docs or export it from our free Incident runbook generator. For a head start, download a Markdown version ready to paste into your wiki:
FAQ
Do we need one runbook per service? Yes. Each critical service needs owners, flags, and commands that fit its failure modes. Keep a short, global overview for company‑wide comms and coordination.
What about multi‑region products? Document region‑local steps and global coordination separately. Failover between regions is one runbook; surviving a single‑region cloud event without paging twice is another.
How detailed should commands be? Detailed enough that a rested engineer can not think for the first five minutes. Under stress, naming the flag and pasting a known command is a gift.