Download the template: DR Runbook Template (PDF)
Most companies that prepare for outages start with an IT disaster recovery plan. That document has a real job to do: it sorts workloads into priority tiers, names who can authorize a failover, and sets recovery targets for the business. However, a governance plan is not sufficient on its own when a primary cloud region or database cluster stops answering at 02:30 UTC. Whoever gets paged still needs exact commands to run, and in the right order.
What happens if an engineer promotes a standby database while the old primary is still reachable from half the application fleet? Or what if the database cutover finishes in ten minutes, only for everyone to realize that public DNS records still carry a one-hour TTL? Maybe the hardest part of a regional failover isn't promoting the replica at all, but rather making sure the impaired primary cannot come back onto the network and split committed writes in two? Closing those execution gaps is what a service-specific disaster recovery runbook is for.
Alongside the PDF template above, you can draft step-by-step procedures in our Operational Maintenance Runbook Generator and build matching drill sheets in our DR Test Plan Generator.
What Belongs in an Executable DR Runbook
A runbook full of bracketed placeholders like <insert-cluster-arn> slows responders down right when the clock is loudest. Fill in literal production identifiers ahead of time: AWS account IDs, cluster ARNs, Route 53 hosted-zone IDs, connection-pooler config paths, and out-of-band credential vault locations. Lower failover DNS TTLs to 30 to 60 seconds permanently as well, so you are never waiting out stale resolver caches during an outage.
Disaster Recovery Runbook Sections and Owners
Service Identification
- Exact production service name, environment, and cloud account
- Primary on-call rotation and escalation contact
- Agreed RTO and RPO targets for this workload
Activation Triggers
- Monitoring conditions that call for failover rather than waiting on repair
- Named role authorized to approve a cross-region promotion
Communications
- Incident channel, out-of-band bridge link, and status-page owner
- Internal and customer status update cadence
Containment and Fencing
- Commands to cut off writes to the impaired primary environment
- Isolation steps for outbound webhooks, billing jobs, and email queues
Restore and Verification
- Exact CLI, SQL, or console recovery steps in execution order
- Application and data consistency checks before reopening traffic
Failback and Follow-Up
- Steps to start backups and WAL archiving in the recovery region
- Observed timings, snags encountered, and runbook edits needed
Walking Through a PostgreSQL Cross-Region Failover
Consider a single-writer SaaS billing service with a 60-second RPO and a 30-minute RTO, backed by an asynchronous cross-region PostgreSQL streaming replica. Promoting the standby itself is quick. Fencing write traffic to the impaired primary before you promote is the step that keeps a brief network heal from leaving you with two active primaries and a messy reconciliation job.
Recovering a Primary Database During a Region Outage
- Page the database on-call engineer, declare the incident, and open the dedicated incident channel and bridge.
- Fence incoming writes at the application or proxy layer using read-only maintenance mode or a durable queue so stray writes cannot hit the old primary.
- Check replication lag in the recovery region, promote
db-replica-1to standalone primary, and record the new writer endpoint. - Update the database connection secret or pooler target and cycle application workers cleanly onto the promoted primary.
- Preload critical cache keys for authentication, configuration, and active catalog queries so cold traffic does not saturate the database.
- Run smoke checks along with row-count or ledger consistency queries before replaying any queued writes.
- Lift read-only mode and watch latency, error rates, and backup health closely before closing the incident.
Replace placeholder hostnames and fencing controls with the exact commands for your stack, and rehearse the full sequence in an isolated environment first.
Before and right after promoting the PostgreSQL standby, run these checks directly on the recovery host to measure any unshipped WAL gap and confirm that recovery mode has cleared:
-- Pre-promotion check on DR standby: verify recovery mode and measure replay lag
SELECT
pg_is_in_recovery() AS in_recovery,
pg_last_wal_receive_lsn() AS receive_lsn,
pg_last_wal_replay_lsn() AS replay_lsn,
CASE
WHEN pg_last_wal_receive_lsn() = pg_last_wal_replay_lsn() THEN INTERVAL '0 seconds'
ELSE now() - pg_last_xact_replay_timestamp()
END AS replay_lag;
-- Promote standby and wait up to 60s for recovery mode to clear (PostgreSQL 12+)
SELECT pg_promote(wait => true, wait_seconds => 60);
-- Post-promotion sanity check: must return false (node now accepts writes)
SELECT pg_is_in_recovery();
Once the standby accepts writes and your connection pooler points to it, pause before opening public traffic all the way. Check the latest committed invoice or order timestamp against your RPO target and record any gap so finance and support know what needs reconciling. Just as important, verify that continuous WAL archiving and base backups are writing from the newly promoted primary into your immutable backup vault. It is not unheard of for a team to pull off a clean cross-region promotion and then run unprotected for two days because backup jobs were only ever wired up in the original region.
And what happens once the original cloud region stabilizes? Resist the temptation to flip traffic straight back. Failing back is a production database migration in reverse, and running it as a scheduled window under your change management plan—seeding a fresh base backup and catching up a streaming replica into the restored home region first—keeps one outage from turning into two.
Keeping Runbooks Current With Scheduled Drills
A DR runbook starts drifting the week Terraform renames a security group, rotates an IAM role, or upgrades a database major version. Give each critical runbook a named service owner and test it on a recurring schedule, using the approach in our DR test plan guide to catch stale commands in an isolated network before a real incident finds them.
Sample Drill Cadence and Responsibilities
| Cadence | Exercise | Owner | What done looks like |
|---|---|---|---|
| Monthly | Isolated backup restore check | Operations lead | Backup restores cleanly in a fenced network and meets target RPO |
| Quarterly | Database replica promotion in staging or isolated VPC | Database or platform lead | Application passes smoke checks against the promoted instance within target RTO |
| Semiannually | Tabletop and multi-service recovery rehearsal | Incident commander and engineering leads | Decision authority, credentials, and comms steps verified; runbook updated |
Adjust drill frequency to match how critical each system is, what your customer contracts require, and how often your infrastructure changes.
If you want an experienced hand setting up cross-region failover, writing service-specific runbooks, or running scheduled recovery drills, take a look at our disaster recovery services and application production support.