Sample postmortem: invoice reminders sent twice
A fictional result of the Incident postmortem recipe: a timeline, two root causes, what went well, and the actions, written without blame.
- Duplicate reminders
- 1,140
- Accounts affected
- 312
- Time to stop it
- 47 min
What happened
- 09:00
Morning reminder run starts
The cron script sends the day's reminders.
- 09:04
Bad: Deploy restarts the worker mid-run
The run had sent about half the batch.
- 09:06
Bad: The script runs again from the start
Nothing recorded which reminders had gone, so all were sent.
- 09:31
Support sees the first replies
Freelancers forwarding confused messages from their clients.
- 09:51
Good: Reminders paused
Sam turns the cron off.
- 11:20
Good: Apology sent to affected accounts
Why
- Sending wasn't idempotent. The script had no record of what it had already sent, so a restart meant starting over.
- Deploys could interrupt a run. Nothing stopped a deploy during the reminder window.
What went well
Support spotted it in under half an hour, and the pause switch worked first time. The apology went out the same morning.
Actions
- Move reminders to the job queue, with one unique key per reminder
- Done: Block deploys from 08:45 to 09:30 until then
- Add an alert when reminders sent in an hour exceed the day's expected count