Template Overview

Every production failure is an expensive educational event paid for by the organization. Failing to capture the systemic lessons from an outage guarantees that the same failure mode will recur. The Post-Incident Action Plan provides a structured blameless retrospective rubric covering minute-by-minute timelines, 5-Whys root cause analysis, detection gap evaluations, and concrete preventive engineering action items.

Markdown Template Structure

Copy the block below or click Download .md File to add this template directly to your personal vault:

---
title: "Incident Retro: [Incident Summary / INC-XXXX]"
incident_date: [YYYY-MM-DD]
retro_lead: "[[Incident Commander / SRE Lead]]"
incident_severity: "[SEV-0 / SEV-1 / SEV-2]"
duration_minutes: [Number]
impacted_customers_percent: [Number]%
tags: [post-mortem, incident-retro, blameless, sre, reliability, root-cause]
---

# Post-Incident Action Plan — [INC-XXXX]

## 1. Incident Overview & Executive Summary
- **Incident Summary:** Audio transcription service failed to initialize on macOS 15 release candidates due to an unexpected Core ML system permission prompt race condition.
- **Total Duration of Disruption:** 42 minutes.
- **Customer Impact:** Approximately 350 beta users experienced transcription initialization timeouts during recording sessions.
- **Data Loss Assessment:** **Zero audio loss.** Raw microphone audio was successfully written to local disk caches; only transcription rendering was delayed.

## 2. Chronological Timeline of Events (UTC)
| Timestamp | Event Description | Detected By | Action Taken |
|---|---|---|---|
| 14:15 | macOS 15 beta seed rolled out to testing cohort | System Update | Users began reporting startup errors |
| 14:22 | Error rate on `neural_init` spiked from 0.01% to 14.5% | Datadog Alert | PagerDuty alerted on-call SRE |
| 14:28 | Incident declared; War room bridge opened | On-Call Lead | Rollback options evaluated |
| 14:36 | Root cause identified: macOS sandbox timing restriction | Tech Lead | Drafted hotfix patch bypassing deprecated API |
| 14:48 | Hotfix binary deployed to auto-updater channel | DevOps | Crash rate dropped back to 0.02% |
| 14:57 | Incident formally resolved; all systems nominal | Commander | War room closed; retro scheduled |

## 3. The 5 Whys Root Cause Analysis
1. *Why did transcription fail to initialize?* Because the Core ML model compilation timed out after 5 seconds.
2. *Why did compilation time out?* Because macOS 15 introduced a new synchronous privacy permission check.
3. *Why did the check block the thread?* Because the initialization ran on the main UI dispatch queue.
4. *Why did it run on the main thread?* Because legacy startup routines grouped permission requests with UI window loading.
5. *Root Cause:* Absence of asynchronous background initialization decoupled from UI thread lifecycles.

## 4. What Went Well vs. What Went Poorly
- **What Went Well:**
  - Zero audio data was lost thanks to local disk audio buffering architecture.
  - SRE on-call was alerted within 7 minutes of initial error spike.
  - The hotfix was built and deployed in under 20 minutes.
- **What Went Poorly:**
  - Automated CI runners did not include macOS 15 beta developer previews.
  - Timeout error message displayed to users was a generic error code rather than an actionable message.

## 5. Preventative Engineering Action Items
| Action Item Description | Type (Prevent / Detect / Mitigate) | Owner | Priority | Target Due Date |
|---|---|---|---|---|
| Decouple neural compilation to asynchronous background worker | Prevent | [[Marcus Vance]] | P0 | [YYYY-MM-DD] |
| Add macOS beta preview runner to GitHub Actions CI test matrix | Detect | [[DevOps Lead]] | P1 | [YYYY-MM-DD] |
| Improve user-facing error dialog with explicit permission guidance | Mitigate | [[Frontend Lead]]| P2 | [YYYY-MM-DD] |

## 6. Closing Blameless Affirmation
- "This retrospective focuses on systemic design gaps, not personal failings. The team executed a swift, disciplined containment under pressure."
- **Retro Completed:** [YYYY-MM-DD]

Filled Real-World Example

Here is a realistic example demonstrating how this template captures actual deliberations:

---
title: "Incident Retro: INC-104 - OAuth Token Expiry During Deployment"
incident_date: 2026-10-09
retro_lead: "[[David Chen]]"
incident_severity: "SEV-2"
duration_minutes: 18
impacted_customers_percent: 0.0
tags: [post-mortem, incident-retro, blameless, sre, reliability, root-cause]
---

# Post-Incident Action Plan — INC-104

## 1. Overview
- Wrangler deployment script failed with `code: 10001 (Unable to authenticate request)` during production CI push.
- Zero customer downtime; live production edge continued serving static cached content without interruption.

## 2. Root Cause
- Wrangler CLI OAuth token expired after 30 days without an automated refresh hook in CI environment.

## 3. Resolution & Prevention
- Generated automated refresh script calling Cloudflare OAuth token refresh endpoint.
- Added pre-flight `wrangler whoami` authentication verification check to deployment workflow.

When to Use and When to Avoid

When to Use

  • Use within 48 hours of any SEV-0, SEV-1, or SEV-2 production outage or significant customer disruption.
  • Use to institutionalize lessons learned and prevent repeat failure modes across engineering teams.
  • Use to build a blameless, psychological-safety-oriented engineering culture.

When to Avoid

  • Avoid using for routine expected exceptions or minor localized unit test failures.
  • Avoid using as a tool for reprimanding or punishing individual engineers.

On-Device AI Integration

Summarize your recording in Oyma with this local instruction:

Analyze the post-incident retrospective action plan:
1. Scrutinize the 5-Whys root cause analysis to verify it uncovered systemic architecture flaws rather than human errors.
2. Check that each preventive action item is concrete, assigned to a single owner, and has a clear priority.
3. Draft a 2-paragraph blameless summary suitable for sharing with all company stakeholders.
Output in structured Markdown.

Your data remains securely on your Mac with zero cloud exposure.

Frequently Asked Questions

What makes an incident retrospective blameless?

It assumes humans had good intentions and worked with the information they had; it investigates systemic and environmental failures rather than assigning personal blame.

Why document incident post-mortems in Markdown?

Markdown post-mortems remain permanently discoverable in git, serving as institutional memory for future on-call engineers.

Can Oyma transcribe post-incident review meetings?

Oyma transcribes incident retro calls locally on Apple Silicon, keeping post-mortem discussions confidential.