# Sample postmortem: invoice reminders sent twice

A fictional result of the Incident postmortem recipe: a timeline, two root causes, what went well, and the actions, written without blame.

> [!info]
> **Fictional example.** Tally and the Fernbrook team are invented. This shows the shape of the blameless postmortem the *Incident postmortem* recipe writes from commits, deploys and notes.

<!-- onecell:block {"type":"stats","items":[{"tone":"bad","label":"Duplicate reminders","value":"1,140"},{"label":"Accounts affected","value":"312"},{"label":"Time to stop it","value":"47 min"}]} -->
| Metric | Value | Change |
| --- | --- | --- |
| Duplicate reminders | 1,140 |  |
| Accounts affected | 312 |  |
| Time to stop it | 47 min |  |

## What happened

<!-- onecell:block {"type":"timeline","entries":[{"body":"The cron script sends the day's reminders.","when":"09:00","title":"Morning reminder run starts"},{"body":"The run had sent about half the batch.","tone":"bad","when":"09:04","title":"Deploy restarts the worker mid-run"},{"body":"Nothing recorded which reminders had gone, so all were sent.","tone":"bad","when":"09:06","title":"The script runs again from the start"},{"body":"Freelancers forwarding confused messages from their clients.","when":"09:31","title":"Support sees the first replies"},{"body":"Sam turns the cron off.","tone":"good","when":"09:51","title":"Reminders paused"},{"tone":"good","when":"11:20","title":"Apology sent to affected accounts"}]} -->
- **09:00** — Morning reminder run starts — The cron script sends the day's reminders.
- **09:04** — Deploy restarts the worker mid-run — The run had sent about half the batch.
- **09:06** — The script runs again from the start — Nothing recorded which reminders had gone, so all were sent.
- **09:31** — Support sees the first replies — Freelancers forwarding confused messages from their clients.
- **09:51** — Reminders paused — Sam turns the cron off.
- **11:20** — Apology sent to affected accounts

## Why

1. **Sending wasn't idempotent.** The script had no record of what it had already sent, so a restart meant starting over.
2. **Deploys could interrupt a run.** Nothing stopped a deploy during the reminder window.

## What went well

Support spotted it in under half an hour, and the pause switch worked first time. The apology went out the same morning.

## Actions

- [ ] Move reminders to the job queue, with one unique key per reminder
- [x] Block deploys from 08:45 to 09:30 until then
- [ ] Add an alert when reminders sent in an hour exceed the day's expected count