125 lines
11 KiB
Markdown
125 lines
11 KiB
Markdown
# A guide to post-mortem meetings and how we run them at incident.io
|
||
|
||
- **期号**: SRE Weekly Issue #397（2023-11-05）
|
||
- **作者**: incident.io
|
||
- **链接**: https://incident.io/blog/a-guide-to-post-mortem-meetings
|
||
|
||
## 简介
|
||
|
||
“Blameless” doesn’t mean no names allowed!
|
||
|
||
> Remember—if discussing the actions of a specific person is being done for the sake of better learning; don’t shy away from it.
|
||
|
||
## 正文
|
||
|
||
October 11, 2023 — 9 min read
|
||
|
||
You've just made it through a particularly tough incident.
|
||
|
||
It was a short outage affecting a subset of customers, so not exactly the end of the world, but bad enough that it involved multiple people across a number of teams to resolve — a step in the [incident response lifecycle](https://incident.io/blog/what-is-the-incident-response-process) that often gets rushed. Either way, the incident was well managed, and the dust has settled.
|
||
|
||
Now what?
|
||
|
||
Most guidance would say that putting together a post-mortem document is a good idea, given the severity of the incident. You've also done this, so what's next?
|
||
|
||
It's time to prepare and run your post-mortem meeting! But it isn't enough to go into a room with the people involved and wish for the best. No, there are a few things that you need to keep in mind before, during, and after your post-incident meeting to get the most out of it.
|
||
|
||
An incident post-mortem meeting is held after an incident — if you want a structured approach, see our [seven-step framework for running incident debriefs](https://incident.io/blog/a-seven-step-framework-for-running-incident-debriefs). **This is a space where a small group of responders and other stakeholders discuss the incident, its contributing factors, and its impacts.**
|
||
|
||
The goal is to go beyond the post-mortem document and discuss the incident in a live setting. It's much less reactive and focused on creating better long-term outcomes.
|
||
|
||
While we're on the topic of post-mortems, it's worth diving into the concept of blamelessness since you'll hear it coming up a lot. If you're looking for a deeper dive, we wrote extensively about the [concept of blameless post-mortems here](https://incident.io/blog/blameless-post-mortems).
|
||
|
||
The thinking is that incidents and any post-incident activities should be free of finger-pointing. Instead of focusing on the "who" behind an incident, blameless culture calls for understanding “why” and “how,” focusing on the many causes of incidents instead—or contributing factors.
|
||
|
||
By doing this, you eliminate the anxiety and stress around post-incident processes.
|
||
|
||
To facilitate a productive post-mortem meeting, it's important to establish a few ground rules. These can help promote open and honest discussion, give everyone equal time to raise any concerns and prioritize addressing critical issues.
|
||
|
||
- **Set the tone of blameless accountability:** Set the tone of blameless accountability: First, creating an environment of blameless accountability is crucial —[psychological safety in incident management](https://incident.io/blog/psychological-safety-in-incident-management) is what makes this possible in practice. Remember—if discussing the actions of a specific person is being done for the sake of better learning; don't shy away from it.
|
||
- **Take notes:** Second, staying on topic and taking good notes during the meeting helps maintain focus and makes capturing important details a bit easier. Pro tip: the person running the meeting should*not* be the person taking notes. Their focus should be as a facilitator.
|
||
- **Set expectations for learning:** Lastly, setting expectations at the beginning of the meeting, emphasizing learning and improvement, and promoting open and respectful discussion can help create a positive and productive environment for everyone.
|
||
|
||
There’s a myriad of ways you can run a post-mortem meeting, but let’s look at some best practices that most folks agree are a good starting point.
|
||
|
||
First things first—it's important to include all relevant stakeholders in the post-mortem meeting, including any incident responders, key decision-makers, and individuals directly involved in the incident. The goal is to make sure that every perspective and nuance is accounted for. Remember, everyone will have a unique take on the incident worth factoring in.
|
||
|
||
Before the meeting, set a clear agenda outlining some of the objectives and topics you want to discuss as a team. This helps keep the meeting focused and ensures that all agreed-upon talking points are covered.
|
||
|
||
It doesn't need to be a rigid agenda, and it's entirely expected (and encouraged!) that you'll end up talking about other topics, but going in with some pre-work and thoughts on areas of interest can be incredibly helpful.
|
||
|
||
Your [post-mortem document](https://incident.io/blog/what-is-a-post-mortem-document) is a good place to start!
|
||
|
||
During the meeting, the incident should be reviewed from end to end, including the timeline of events, the impact on the organization, and the actions taken to mitigate the incident. This allows the team to gain a complete understanding of what happened.
|
||
|
||
Discussion should be actively encouraged during the review. Remember, the goal is to learn, not recite the contents of the document. If people are less forthcoming with contributions, it can be helpful to ask open questions to gather additional perspectives.
|
||
|
||
Next, try to identify any contributing factors to the incident. This involves analyzing technical issues, human error, or process gaps. By identifying these, teams can implement process and technical improvements to help prevent similar incidents in the future.
|
||
|
||
The post-mortem meeting should provide an opportunity for open and honest discussions about the lessons learned from the incident. This includes identifying areas for improvement, discussing best practices for resolving incidents like these, and sharing knowledge among team members.
|
||
|
||
Based on the insights gained from the meeting, a follow-up plan should be created to address the identified issues and prevent future incidents. This plan should include specific tasks, owners, and timelines.
|
||
|
||
It can be tempting to create long lists of actions in a post-mortem meeting — understanding [why post-mortem action items die](https://incident.io/blog/why-post-mortem-action-items-die) before you write them down helps you commit only to the ones that will actually survive. At the point of reviewing this incident, these actions are likely to feel incredibly important–this is [recency bias](https://en.wikipedia.org/wiki/Recency_bias) at play!
|
||
|
||
To counter this, we’d encourage you to avoid *committing* to actions until the owning team has had time to understand the plans and weigh up the relative importance against other planned work.
|
||
|
||
Here at [incident.io](https://incident.io/), we run a debrief meeting (our name for a post-mortem) for every major or critical incident that we respond to — whether it originated in our [incident response tools](https://incident.io/incident-response-slack) or surfaced through another channel. We also occasionally run them for lower-severity incidents too, at the discretion of the incident lead or by request from anyone involved.
|
||
|
||
For less severe incidents, it's common for the lead to write the postmortem and run the debrief. For bigger incidents, someone else, like a tech lead, will run the debrief. This frees up other key players to contribute to the conversation.
|
||
|
||
We also record the meetings using Google Meet, so they’re available for future reference and to share with anyone interested. This is a great way for folks to benefit from hearing the conversation in the room, without needing to inflate the invite list.
|
||
|
||
To make it easier to prepare for the meeting, we use the [post-mortem template](https://incident.io/blog/incident-post-mortem-template) built into incident.io, which pulls specific information from the incident into a debrief document that we use to steer our conversation.
|
||
|
||
That said, we don't require a super polished post-mortem document before running the debrief. One of our company values is to [trust by default](https://incident.io/blog/trust-should-not-start-at-zero), so we let folks choose what’s important to include and go from there.
|
||
|
||
When going through the meeting, this is the structure we usually stick with:
|
||
|
||
1. The facilitator walks through the post-mortem document, i.e., briefly summarizes the incident and walks through the timeline.
|
||
2. The discussion happens, normally based on a set of "talking points" or "learning points."
|
||
3. New follow-up actions are created during the discussion to make sure we’re applying our learnings and building better resilience.
|
||
4. After the meeting, the follow-ups are completed or pushed into an issue tracker, and the notes are sent around to all relevant stakeholders
|
||
|
||
We're good at stopping ourselves from going into rabbit holes or overrunning during these meetings. If it looks like something needs to be investigated or a particular problem needs to be solved, we'll create a follow-up for someone to keep the session focused on learning.
|
||
|
||
But we also make sure that we aren’t ever “overcorrecting” either. If we’re setting any follow-up actions, we aim for them to be proportionate to the severity of the incident.
|
||
|
||
To round things out, we’ve compiled a few tips for running post-mortem meetings from two of our incident responders here at incident.io, [Milly](https://incident.io/blog/author/milly) and [Sam](https://incident.io/blog/author/sam).
|
||
|
||
- Don't invite too many people to the post-mortem meeting. If it's valuable for people to "sit in," record it instead. The more people attend, the harder it is to prompt honest discussion.
|
||
- The same goes for leadership. If you invite all senior leaders, the environment will feel more like "this really bad thing happened, and all the leaders are watching you explain yourself."
|
||
- There are two ways you can learn from what happened during an incident:
|
||
- How can we stop this kind of problem from happening again in the future?
|
||
- How can we respond to this kind of problem better next time?
|
||
- Don't wait too long to have the debrief. It's better to have a scrappier post-mortem and to hold the debrief sooner than it is to polish the doc for ages and have forgotten what happened by the time you run it. Remember, you can always edit it after the meeting.
|
||
|
||
In the end, what we find most important is having the space to discuss the incident at length. The goal is always to come out of the meeting with a shared understanding of how an incident happened and an agreement on whether there are any follow-up actions we need to take—and who’s doing them.
|
||
|
||
For us, this is what a successful post-incident meeting looks like.
|
||
|
||
Incident post-mortems are a crucial document that cannot be glossed over. In this article, you’ll find our go-to post-mortem template that you can use in your own organization.
|
||
|
||
Chris Evans
|
||
|
||
April 16, 2025
|
||
|
||
Post-mortem documents are a great way to facilitate learning after incidents are resolved.
|
||
|
||
Luis Gonzalez
|
||
|
||
September 27, 2023
|
||
|
||
While blameless post-mortems are a great idea on the surface, if taken to the extreme, they can muddy how much you actually learn from incidents.
|
||
|
||
incident.io
|
||
|
||
October 4, 2023
|
||
|
||
Ready for modern incident management? Book a call with one of our experts today.
|
||
|
||
- All-in-one incident management
|
||
- Our unmatched speed of deployment
|
||
- Why we’re loved by users and easily adopted
|
||
- How we work for the whole organization
|