Production Incident Post-Mortem & RCA

Template Overview

Complex distributed systems and native desktop applications inevitably fail. What distinguishes elite engineering organizations from fragile ones is not the total absence of incidents, but how thoroughly the team learns from every failure to ensure it never happens again.

A blameless Incident Post-Mortem analyzes what failed without pointing fingers. It examines system boundaries, monitoring gaps, architectural assumptions, and alerting thresholds.

This template provides an industry-standard post-mortem framework: incident severity classification, customer blast radius, chronological timeline of events, 5 Whys root-cause analysis, and prioritized preventive action items.

Markdown Template Structure

Copy the block below or click Download .md File to add this template directly to your personal vault:

---
title: "Incident Post-Mortem: [Incident ID / Title]"
date: [YYYY-MM-DD]
incident_id: "INC-2026-92"
severity: P1 - Degraded Service
incident_commander: "[[Staff SRE Name]]"
author: "[[Lead Systems Engineer]]"
reviewers:
  - "[[Chief Architect]]"
  - "[[Head of Product]]"
incident_start: "2026-10-06T11:00:00Z"
incident_resolved: "2026-10-06T12:35:00Z"
total_downtime_minutes: 95
tags: [incident, post-mortem, rca, reliability, engineering]
---

# Incident Post-Mortem: [Incident ID / Title] — [YYYY-MM-DD]

## 1. Executive Summary & Customer Impact
- **What Happened:** Users running long audio recordings on older M1 Mac hardware experienced sporadic audio frame drops and transcript stuttering.
- **Customer Blast Radius:** Affected approximately 120 users (3.2% of active desktop sessions during the incident window).
- **Data Loss:** Zero permanent data loss. Audio recorded to disk buffer was completely recovered following app restart.

## 2. Chronological Incident Timeline (All times in UTC)
| Timestamp | Event / Observation | Actor / System |
| :--- | :--- | :--- |
| 11:00 | First customer bug report submitted regarding audio stutter | Customer Support |
| 11:15 | Incident declared; War Room bridge convened | Incident Commander |
| 11:35 | Reproduced issue on baseline M1 Mac Mini running heavy compile jobs | Systems Eng |
| 11:55 | Identified root cause: CoreAudio ring buffer allocated with 256 samples was too tight | Systems Eng |
| 12:15 | Hotfix PR #830 created, increasing ring buffer to 1024 samples with dynamic backpressure | Systems Eng |
| 12:30 | Canary build deployed to beta ring; zero frame drops confirmed under thermal stress | QA / Eng |
| 12:35 | Incident officially declared resolved; public status page updated | Incident Commander |

## 3. Root-Cause Analysis (5 Whys)
1. **Why did audio frames drop?** The audio callback buffer overflowed during peak processing spikes.
2. **Why did the buffer overflow?** The ring buffer was statically sized at 256 samples, leaving insufficient margin for thread scheduling latency.
3. **Why was the buffer sized at 256 samples?** It was originally optimized for ultra-low latency on M3 Max development machines.
4. **Why wasn't this caught in automated testing?** Automated CI/CD integration tests ran only on high-end hardware without thermal throttling simulation.
5. **Why wasn't low-spec hardware tested?** We lacked a dedicated low-spec hardware testing tier in our automated test pipeline.

## 4. What Went Well and Where We Got Lucky
- **What went well:** Rapid reproduction of the bug within 20 minutes of report triage.
- **Where we got lucky:** Zero audio disk buffers were corrupted; complete transcript integrity was preserved.

## 5. Preventive Action Items & JIRA Backlog
> [!IMPORTANT] Remediation Commitments
> - [ ] [[Systems Eng]] Add automated buffer backpressure telemetry for local warning logs.
> - [ ] [[Lead SRE]] Add older M1 base hardware into nightly automated stress test harness.
> - [ ] [[Lead Architect]] Document CoreAudio buffer sizing constraints in architecture wiki.

Filled Real-World Example

Here is a realistic example demonstrating how this template captures actual deliberations:

---
title: "Post-Mortem: Search Index Lock Contention"
date: 2026-10-07
incident_id: "INC-2026-88"
severity: P2 - Performance Degradation
author: "[[Alex Rivera]]"
tags: [incident, post-mortem, search]
---

# Post-Mortem: Search Index Lock Contention — October 7, 2026

## 1. Summary
Search queries took up to 1.4 seconds on vaults with >50,000 notes due to concurrent WAL compaction locks.

## 2. Root Cause
Background SQLite auto-vacuum executed synchronously on the main thread during heavy note updates.

## 3. Preventive Action
- Separated vacuuming to background GCD queue during idle system state.
- Added automated benchmark alert if search query exceeds 50ms.

When to Use and When to Avoid

When to Use

• Use following production outages, severe software bugs, or security incident resolutions.
• Use to establish a transparent, blameless chronological timeline of operational failures.
• Use to track prioritized engineering action items that permanently eliminate recurring vulnerabilities.

When to Avoid

• Avoid using during the live crisis response phase (use [[crisis-response-war-room]] instead).
• Avoid using for minor styling or visual cosmetic bugs.

On-Device AI Integration

Generate production post-mortems with Oyma on macOS:

Analyze the post-mortem debrief audio transcript:
1. Synthesize the executive summary, customer impact scope, and downtime duration.
2. Compile the chronological timeline of incident events with exact UTC timestamps.
3. Formulate the 5 Whys root-cause analysis tracing from symptoms to organizational causes.
4. Document what went well, where the team got lucky, and concrete preventive action items.
Output in blameless, engineering-grade CommonMark.

Your proprietary infrastructure vulnerabilities and incident post-mortems remain strictly on your Mac.