Incident postmortem
Timeline, causes and actions.
- For
- Whoever was on call, or whoever fixed it
- When
- The day after, while it is fresh
- Takes
- A few minutes of agent time
- You get
- A postmortem with a timeline, root causes, the key fix, what went well and actions, ticking those already done
Write a postmortem for <incident> in onecell. Save it to <cell>. 1. Gather the facts: the commits and pull requests that fixed it (git log, gh pr view), deploy times, any parked debugging session or handoff (search), and what I tell you. Mark any time you estimate as approximate. 2. Draft "Postmortem: <incident>" with create_document (dry run first): - A stats block: duration, impact, fixes - A summary in plain words: what people saw, and why - A timeline block: detection, each finding, each fix, recovery - Root causes as a table: cause, why it happened, why tests missed it - A diff block of the key fix, trimmed - What went well - Actions as a checklist, ticking the ones already done Keep it blameless: describe what the system allowed, not who slipped. 3. If a fix carried a design decision, record it as a decision block too. 4. Reply with the link and the one action that most prevents a repeat.
What your agent does
Reconstructs the timeline
From commit times, pull requests, deploys and your notes, marking what is estimated.Separates causes from symptoms
Each root cause with why it happened and why tests didn't catch it.Shows the fix
The key change as a before-and-after diff, trimmed to the lines that matter.Turns it into actions
A checklist, with the fixes already shipped ticked, and the one that matters most in the reply.
What you get back
Open the full example →A postmortem for Tally, a made-up invoicing app, whose reminder emails went out twice after a deploy restarted the worker mid-run. Two root causes, one already patched.
Sample postmortem: invoice reminders sent twice
- Duplicate reminders
- 1,140
- Accounts affected
- 312
- Time to stop it
- 47 min
What happened
- 09:00
Morning reminder run starts
The cron script sends the day's reminders.
- 09:04
Bad: Deploy restarts the worker mid-run
The run had sent about half the batch.
- 09:06
Bad: The script runs again from the start
Nothing recorded which reminders had gone, so all were sent.
- 09:31
Support sees the first replies
Freelancers forwarding confused messages from their clients.
- 09:51
Good: Reminders paused
Sam turns the cron off.
- 11:20
Good: Apology sent to affected accounts
Why
- Sending wasn't idempotent. The script had no record of what it had already sent, so a restart meant starting over.
- Deploys could interrupt a run. Nothing stopped a deploy during the reminder window.
What went well
Support spotted it in under half an hour, and the pause switch worked first time. The apology went out the same morning.
Actions
- Move reminders to the job queue, with one unique key per reminder
- Done: Block deploys from 08:45 to 09:30 until then
- Add an alert when reminders sent in an hour exceed the day's expected count
Variations
Park a debugging session while it's happening
During the incident, every so often say: “Park what we know so far in my Inbox: hypotheses, commands tried, what each showed.” It takes seconds, survives a crashed session or a switch of agent, and becomes the postmortem's best source.
A note for customers
Add: “Then write a short customer-facing note: what they saw, when it was fixed, and what we changed. No internal names.” Share it with a link after you have read it.
A near miss
Use the same prompt for something caught before anyone noticed. Near misses are the cheapest postmortems you will ever write.
Tips
- Write it the day after. Details fade faster than you expect.
- “Why tests missed it” is the column that prevents the next one.
- Tick the actions already shipped; it shows the loop closing.