Disaster Recovery Runbook

Template Overview

When catastrophic outages, ransomware incidents, or cloud region failures occur, engineering teams cannot rely on cloud wikis or web dashboards that may be part of the outage blast radius. The Disaster Recovery Runbook provides infrastructure leads and SRE teams with an air-gapped, verifiable recovery plan in Markdown. It outlines step-by-step failover commands, communication escalation trees, database restore procedures, and post-restoration verification checks.

Markdown Template Structure

Copy the block below or click Download .md File to add this template directly to your personal vault:

---
title: "DR Runbook: [System / Service Name]"
date: [YYYY-MM-DD]
sre_lead: "[[SRE Lead Name]]"
infrastructure_owner: "[[VP Engineering]]"
target_rto_hours: 2.0
target_rpo_minutes: 15.0
disaster_tier: "[Tier 0 Mission-Critical / Tier 1 Core]"
tags: [disaster-recovery, sre, devops, runbook, infrastructure, incident-response]
---

# Disaster Recovery Runbook — [System / Service Name]

## 1. System Topology & Recovery Targets
- **Primary Region:** AWS `us-east-1` (Primary Data Center)
- **Secondary DR Region:** AWS `eu-west-1` (Warm Standby)
- **Recovery Time Objective (RTO):** 2 hours from formal declaration.
- **Recovery Point Objective (RPO):** Maximum 15 minutes of asynchronous replication lag.
- **Failover Authority:** Any designated Incident Commander or On-Call Lead.

## 2. Emergency Incident Command & Escalation Tree
| Role | Primary Contact | Secondary Contact | Escalation SLA | Secure Comms Channel |
|---|---|---|---|---|
| Incident Commander | [[Incident Commander]] | [[Deputy IC]] | 5 minutes | Signal Group `DR-Command-Alpha` |
| Database SRE | [[Database Lead]] | [[SRE On-Call]] | 10 minutes | Secure Phone Bridge |
| Communications Lead | [[VP Marketing]] | [[Head of Support]] | 15 minutes | External Statuspage Admin |

## 3. Step-by-Step Failover Sequence
### Phase 1: Assessment & Outage Confirmation (0-15 mins)
```bash
# 1. Verify regional outage status
aws health describe-events --region us-east-1

# 2. Check primary database replication lag
pg_isready -h db-primary.internal -p 5432

Phase 2: DNS & Traffic Rerouting (15-30 mins)

# Reroute Cloudflare Edge Workers / Route53 failover records
curl -X POST "https://api.cloudflare.com/client/v4/zones/$ZONE_ID/dns_records/$FAILOVER_ID"   -H "Authorization: Bearer $CF_DR_TOKEN"   -d '{"content": "dr-ingress.eu-west-1.example.com"}'

Phase 3: Database Standby Promotion (30-60 mins)

# Promote replica in secondary region to primary read/write
ssh sre-bastion.eu-west-1
pg_ctl promote -D /var/lib/postgresql/data

4. Integrity Verification & Sanity Checks

• Database read/write health check passes with sub-50ms latency.
• Authentication service correctly validates active JWT tokens.
• Real-time ingestion queues accept payloads without dead-letter backpressure.
• Public status page updated to: “Systems Operational via DR Secondary”.

5. Failback & Post-Restoration Protocol

• Do NOT initiate failback until primary datacenter has maintained 4 continuous hours of zero-fault stability.
• Reverse replication stream before migrating traffic back to primary region.

## Filled Real-World Example

Here is a realistic example demonstrating how this template captures actual deliberations:

```markdown
---
title: "DR Runbook: Primary Authentication & Storage API"
date: 2026-10-09
sre_lead: "[[David Chen]]"
infrastructure_owner: "[[Elena Rostova]]"
target_rto_hours: 1.0
target_rpo_minutes: 5.0
disaster_tier: "Tier 0 Mission-Critical"
tags: [disaster-recovery, sre, devops, runbook, infrastructure, incident-response]
---

# Disaster Recovery Runbook — Primary Authentication & Storage API

## 1. System Topology & Targets
- Target RTO: 60 minutes.
- Target RPO: 5 minutes.
- Failover target: Standby cluster in Frankfurt datacenter.

## 2. Failover Sequence
1. Confirm regional failure via external health ping monitors.
2. Promote Postgres read-replica to primary writer (`pg_ctl promote`).
3. Switch Cloudflare DNS CNAME to point to Frankfurt edge worker.
4. Verify HTTP 200 on `/health/ready` endpoint.

## 3. Verification Checklist
- [x] Verified zero database split-brain conditions.
- [x] Confirmed customer token validation latency < 25ms.
- [x] Updated internal incident Slack channel and external status page.

When to Use and When to Avoid

When to Use

• Use during major datacenter migrations, catastrophic cloud outages, or ransomware isolation drills.
• Use to preserve an offline, air-gapped recovery plan that does not depend on cloud uptime.
• Use to conduct quarterly chaos engineering and disaster simulation fire drills.

When to Avoid

• Avoid using for minor individual server restarts or simple application bug patches.
• Avoid using as a routine customer support troubleshooting guide.

On-Device AI Integration

Summarize your recording in Oyma with this local instruction:

Analyze the disaster recovery runbook:
1. Verify the logical order of failover commands to ensure database integrity is preserved before DNS rerouting.
2. Check for missing safety checks that could cause split-brain data corruption.
3. Formulate a tabletop simulation scenario to test the engineering team against this runbook.
Output in structured Markdown.

Your data remains securely on your Mac with zero cloud exposure.