A disaster recovery runbook documents how to declare an incident, contain the failure, fail over, verify recovery, and restore normal operation. This guide includes a template to adapt, a primary database outage example, and a schedule for testing the procedure.
What to Include in a DR Runbook
A runbook is the agreed path the team follows under pressure. It names owners, flags, commands, and checks. It does not try to predict every cause or replace engineering judgment. Document the initial response for the failure types your service faces, such as region loss or storage corruption.
Disaster Recovery Runbook: Sections and Owners
Identification
- Service name – exactly as operators know it
- Primary owner – on‑call rotation or team
- Criticality – user impact and RTO/RPO
Triggers
- Which alerts declare this runbook active
- Manual triggers (executive call, legal, safety)
Communications
- Where incidents live (channel, bridge, ticket)
- Stakeholder updates: who, when, and format
Mitigation Steps
- Immediate safety checks (kill switches, flags)
- Containment actions to limit blast radius
- Failover / restore procedures with commands
Recovery & Verification
- Back to steady state: how we know
- Data integrity checks and reconciling backlogs
- Roll back temporary mitigations
Aftercare
- Follow‑ups, owners, and due dates
- What to improve in this runbook
Template Sections With Field‑by‑Field Guidance
Use the sections above to document exactly how this service fails over and recovers. Copy the headings into your docs and fill them with the concrete names your team uses: on‑call rotation, feature flags, queue names, dashboards, and verification commands. Resolve missing or unclear instructions before an incident occurs.
Filled Example: Recovering a Primary DB in a Region Outage
This example shows the minimum useful level of detail. It does not depend on one vendor or a single product; adapt the names and commands to match yours.
Recovering the Primary Database During a Region Outage
- Declare the incident. Page database on‑call. Open
#inc‑db‑outageand a bridge. - Contain write traffic. Flip
WRITE_READS_ONLYflag to route writes to a queue; protect idempotency keys. - Promote replica. In the standby region, promote
db‑replica‑1to primary and record the new endpoint. - Redirect apps. Update the secrets/connection string and rotate app instances to pick up the change.
- Warm critical caches. Preload product, pricing, and account caches to reduce cold‑start pressure.
- Verify integrity. Run consistency checks; reconcile the write queue into the new primary.
- Lift mitigations. Remove
WRITE_READS_ONLYand close the incident once SLOs hold for one hour.
Adapt the steps, flags, and service names to your environment, then test the procedure before using it during an outage.
Testing the Runbook: Tabletop and Time‑Boxed Drills
Schedule recurring exercises: a quarterly tabletop to surface owners and decisions, a monthly DB failover in staging with a hard stop, and a weekly backup restore to prove RPO.
Keep the Runbook Current: Drills and Owners
| Cadence | Exercise | Owner | Success |
|---|---|---|---|
| Quarterly | Tabletop a cloud region loss | Infra on‑call lead | Roles, flags, and comms surfaced in 15 min |
| Monthly | Time‑boxed DB failover in staging | DBA / app owner | Service healthy on new primary in < 20 min |
| Weekly | Restore a named backup | Ops | Checksum verified; RPO within target |
Schedule the exercises and name an owner for each check. Update the runbook with any missing steps or incorrect assumptions found during a drill.
Keeping It Current: Owners, Cadence, and Change Hooks
Assign an explicit owner for this document and connect it to change. When a team adds a dependency, ships a new feature flag, or rotates a secret, the runbook should change in the same pull request. Add a question to the PR template asking whether the change affects disaster recovery.
Downloads
You can copy this template into your docs or export it from our free Incident runbook generator. For a head start, download a Markdown version ready to paste into your wiki:
FAQ
Do we need one runbook per service? Yes. Each critical service needs owners, flags, and commands that fit its failure modes. Keep a short, global overview for company‑wide comms and coordination.
What about multi‑region products? Document region‑local steps and global coordination separately. Failover between regions is one runbook; surviving a single‑region cloud event without paging twice is another.
How detailed should commands be? Include the exact commands, required access, expected output, and conditions for using them. Test the instructions with an engineer who did not write them.