Pindot
Draft — pending review

Writing a postmortem the review can trust

A postmortem is only as good as the evidence under it. How to build the timeline from records, separate cause from contributing factors, and leave actions that get done.

Pindot team

Incident response4 min read

On this page
  1. Start from records, not memory
  2. Separate the trigger, the cause and the contributing factors
  3. Write it blamelessly, and still be specific
  4. Actions with owners and dates
  5. Attach the evidence
  6. A template that keeps you honest
  7. Review the review
  8. Where Pindot helps

The review meeting after an incident has one job: to make the next incident less likely or less painful. It can only do that if the document in front of it is accurate. Too many postmortems are written from memory a day later, shaped by whoever spoke loudest on the bridge, and quietly avoid the uncomfortable parts. This guide is about writing one that the room can trust.

Start from records, not memory

Build the timeline from things that were recorded at the time: alerts, log lines, change records, deployment history, chat messages with their timestamps, and the scribe's notes from the bridge. Memory is useful for filling in why someone made a decision, but it is unreliable about when, and about the order of events.

  • Give every entry a time and a source. If an entry has no source, mark it as recalled rather than recorded.
  • Use one time zone throughout, and say which.
  • Include the moments when nothing happened, such as a 25-minute gap while the wrong team investigated. Those gaps are often the most useful part of the review.

Separate the trigger, the cause and the contributing factors

Most incidents have more than one thing worth fixing. Separating them stops the review from settling on one convenient explanation.

  • The trigger is what set it off: a release, a traffic spike, a certificate expiring.
  • The cause is why the trigger led to harm: for example, a configuration change that cut the database connection pool to a fifth of what peak traffic needed.
  • Contributing factors are why it lasted as long as it did, or hurt as much: an alert that fired late, a change rated low risk, a runbook that pointed at the wrong dashboard.

Write it blamelessly, and still be specific

Blameless does not mean vague. It means describing decisions in the context in which they were made, with the information that person had at the time. The change was rated low risk because the change form had no field for connection limits is blameless and specific. The engineer should have known better is neither.

Write about systems, steps and information, not about character. Name roles rather than people where you can. If the process allowed a mistake to reach production, the process is the finding.

Actions with owners and dates

An action without an owner is a wish. Every action in the document should have one named owner, a due date, and a link to the ticket where the work is tracked. Group actions by what they do, so the review can see whether the plan is balanced.

  • Prevent: stop this cause from happening again, such as validating pool limits in the release pipeline.
  • Detect: notice sooner next time, such as alerting on connection wait time, not only on errors.
  • Mitigate: reduce harm when it does happen, such as a documented, rehearsed rollback for configuration changes.

Be wary of actions that only say be more careful or improve communication. If an action cannot be checked as done, rewrite it until it can.

Attach the evidence

A reader should be able to check any claim in the document without asking the author. Link or attach the records each finding rests on.

  • The alerts that fired, with their times, and any that should have fired but did not.
  • The log lines that show the failure, quoted exactly.
  • The change record or deployment that triggered it.
  • The bridge timeline as the scribe wrote it.
  • Impact measures: how long, how many users or transactions, which services.

A template that keeps you honest

Postmortem outline
Summary         Two or three sentences a newcomer can follow
Impact          Who was affected, how, and for how long
Timeline        Time, event, source of the record
Trigger         What set it off
Cause           Why the trigger led to harm
Contributing    Why it lasted, or hurt, as much as it did
What went well  Worth keeping next time
Actions         Owner, due date, ticket, and prevent/detect/mitigate
Evidence        Links to every record cited above

Review the review

A month later, look back at the actions. Which were done, which were quietly dropped, and did any of them change what happened in the next incident? The postmortem process improves the same way systems do: by looking at what actually happened, not at what was planned.

Where Pindot helps

Much of the effort in a postmortem goes into finding the records again. When an incident was investigated with Pindot, every answer it gave already cites the exact records it used, and leaves an audit record of where it looked and what it found. That gives the author a timeline with sources from the first draft. The judgement about causes and actions stays with your team.

Stop digging through logs at 2am.

Bring your own 02:14, a real incident from your systems. We will show you how Pindot finds the cause, and where every piece of evidence came from.