A postmortem is easy to hold and hard to make useful. The meeting happens, a document is written, four action items are recorded, and six months later the same failure occurs and somebody finds the old document while searching for something else.
The word “blameless” gets most of the attention in discussions of this, and it is genuinely important, but it is not where the failure is. Plenty of organisations run scrupulously blameless reviews that change nothing, because the review was treated as the deliverable rather than as the thing that produces one.
The test of whether yours are working is embarrassingly simple: what proportion of action items from the last six months are done? If the answer is most of them, the process is working whatever it looks like. If it is under half, you are holding meetings rather than running a process, and adding better templates will not fix it. Fewer items, owned and scheduled, beats a thorough list nobody has time for.
What a Postmortem Is Actually For
Not to explain what happened. The people in the room usually know that by the end of the incident.
It exists to find the conditions that allowed a small problem to become a large one, and to change some of them. That is a different question from what broke, and the distinction shows up immediately in the write-up. “The database ran out of connections” is what broke. “Nothing alerted on connection pool saturation, and the dashboard that would have shown it is not on the on-call runbook” is a condition, and it is actionable.
The second thing it produces, which is undervalued, is a shared account of the event. Without one, everybody keeps their own version, and those versions diverge in the direction of whatever each person already believed about the system.
Blameless Means Something Specific
It does not mean nobody made a mistake. It means the analysis assumes that a person acting reasonably on the information available took an action that turned out to be wrong, and asks why that action looked correct at the time. Google’s SRE book sets out the practice and the failure modes that come with it.
The reason this is not merely kindness is that blame destroys information. In an organisation where incidents attach to individuals, people describe events carefully, omit the detail that looks bad, and delay reporting until they understand it well enough to present it safely. The review then works from an account that has been edited for self-protection, which is the worst possible input.
Practically, blameless is a property of language. “Why did you restart the service?” invites defence. “What did the dashboard show when you decided to restart?” invites information. The second question is also the more useful one, because the answer is a fact about your monitoring rather than about a colleague.
If a genuine performance problem exists, address it separately and directly. Doing it inside the incident review corrupts the review and is, in any case, a poor way to have that conversation.
Building a Timeline That Teaches
The timeline is the most valuable part of the document and the part most often written badly, as a sequence of technical events with the humans removed.
Include what people knew and when. Not just “the service was restarted at 14:32”, but what the person could see at 14:32 and what they believed was happening. That is where the actionable material is, because a fifteen-minute delay caused by an ambiguous dashboard is a fixable problem and a fifteen-minute delay recorded as a gap is not.
Mark the moments that mattered: when it started, when anybody first noticed, when the right person was involved, when the cause was understood, when it was mitigated, when it was resolved. The gaps between those are where the improvement lives, and detection time is very often the largest one.
Write it while it is fresh, ideally within a day or two. Memory of what somebody believed at a particular moment decays much faster than memory of what happened.
There Is Rarely One Root Cause
The phrase encourages stopping at the first plausible explanation, and incidents of any size are chains where several things had to hold simultaneously.
A useful review lists contributing factors instead: the change that triggered it, the test that would have caught it and did not exist, the alert that fired to a channel nobody watches, the runbook that was accurate for a previous architecture. Each is a separate opportunity, and you can choose to fix the cheapest two rather than the one that happened to be first in the chain.
This also reduces a common distortion. When a review names one root cause, that cause is very often “human error”, which is a description rather than an explanation and closes the investigation exactly where it should have started.
Making the Follow-Up Happen
This is where postmortems live or die, and the fixes are unglamorous.
Fewer items. Two that get done beat eight that do not. Rank them by whether they reduce detection time, blast radius or likelihood, and cut the rest.
A named owner and a date, in your normal work tracker rather than in the document. Items that live only in the postmortem are invisible during planning, which is when work actually gets scheduled.
Review the open items at the next postmortem. This single habit does more than any template. It makes incompletion visible and it turns the sequence of reviews into a process rather than a series of unrelated meetings.
Accept some items honestly. A team that consistently completes half of what it commits to should commit to half as much. Recording an intention nobody will fund is worse than recording a decision not to act, because the second is at least true.
The improvements that come out of this are usually the same family covered in our disaster recovery guide and in software testing strategies: faster detection, smaller blast radius, and a system somebody other than the original author can operate.
Mecanik runs incident reviews on systems we maintain as part of our server security work. The pattern across most of them is that the technical cause was understood quickly and the detection gap was the expensive part.
Frequently Asked Questions
What is the point of a postmortem? To find the conditions that let a small problem become a large one and change some of them, which is a different question from what broke. It also produces a shared account of the event, without which everyone keeps a private version that drifts toward what they already believed about the system.
What does blameless actually mean? That the analysis assumes a person acting reasonably on the information available took an action that turned out wrong, and asks why it looked correct at the time. It matters because blame destroys information: people edit their accounts for self-protection and the review then works from a corrupted input.
Should a postmortem identify a single root cause? No. Incidents of any size are chains where several conditions had to hold at once, so listing contributing factors is more useful. It lets you fix the cheapest two rather than whichever came first in the chain, and it avoids concluding with “human error”, which is a description rather than an explanation.
What should an incident timeline include? What people knew and when, not only what the systems did. Record when it started, when anybody first noticed, when the right person was involved, when the cause was understood, when it was mitigated and when it resolved. The gaps between those are where improvement lives, and detection time is usually the largest.
Why do postmortem action items never get done? Because they live in the document rather than in the work tracker, so they are invisible when work is scheduled. Fix it with fewer items, a named owner and a date in the normal tracker, and a review of open items at the start of the next postmortem, which makes incompletion visible.
Comments