Skip to content

Post-Mortems

A post-mortem is a written review of an incident. Its purpose is to find out what happened, why it was possible, and what changes so that it does not happen again. It is not a search for someone to blame.

Write one for every severity 1 incident, every major incident (Incident management), and for any other incident where the team learned something worth sharing. A reasonable target is a draft within five working days. Share it with the affected customer or workload teams. In the managed model the platform team (XMMS) writes it and the customer reviews it. In the self-managed model your own platform team writes it for its stakeholders.

Principles

  • Blameless. Describe what people knew and decided at the time. Ask what in the process or tooling allowed the mistake.
  • Facts first. A time-stamped timeline before conclusions. Use one time zone and say which.
  • Root cause and contributing factors. There is rarely one cause. Name the trigger and the conditions that made it possible or made it worse.
  • Actions have an owner and a date. An action without an owner does not happen. Track the actions in the same backlog as other platform work, and review them in the operational meeting.
  • Keep customer-identifying data out of the shared copy (names of people, IDs, internal links). Use roles and resource types.

Impact scale

Use a fixed scale so reports can be compared. An example:

Impact Meaning
Critical No working connection or service for production. Several applications unusable
Severe / High A production application or service is unusable, or a resource was destroyed or lost data
Medium Short interruptions that were visible in logs, but no complaints from users
Low No visible impact. Near miss

Template

Copy this into a new page, one page per incident. Name it <date> | <short description>.

# <date> | <short description>

## Summary
Two to five sentences: what happened, what was the impact, how it was fixed.

## Impact
Level (see the impact scale), who was affected, for how long.

## Timeline
All times in <time zone>.

| Time | What happened | Who |
|---|---|---|
| hh:mm | Incident reported / detected | reporter / monitoring |
| hh:mm | ... | ... |
| hh:mm | Service restored | ... |

## Root cause
The trigger, and why the platform or the process allowed it. If the cause is not
confirmed, say so and say what is assumed.

## What went well
What worked and should be kept.

## Learnings
What the team would do differently, and what it did not know.

## Actions
| Action | Owner | Due | Status |
|---|---|---|---|
| ... | ... | ... | Open / Done |

Example lessons

The lessons below are generalised from post-mortems of real incidents on managed platforms. They are examples of the format and of the kind of lesson that is worth writing down. The details, including names and numbers, have been removed.

A WAF setting caused 403 errors on production applications
Impact Severe: several production applications were unreachable behind the Application Gateway
Trigger Two days earlier a WAF setting (inspection of the request body) was switched on, because a documentation review asked why it was off and enabling it looked like an obvious hardening step. It was not tested, and no change record was made
Detection A user reported 403 Forbidden. The WAF logs showed matched rules
Fix The matched rules were disabled temporarily. The setting was then switched off, and the rules were turned on again. About three hours from report to resolution
Contributing factors No test environment for WAF changes. The customer was not told about the change. The incident was first treated as a service request and not as a P1, although applications were down

Lessons:

  • Test WAF and firewall changes on the canary platform environment first, with real traffic patterns, before prod. See Application Gateway and Change workflow.
  • Every change to the platform has a change record, also a "small" setting. Unannounced changes make incidents harder to diagnose.
  • Register an incident with production impact as a P1, whatever the first report says.

A large firewall change hit the apply time-out, and the fix made it worse
Impact Medium: short traffic interruptions visible in firewall logs, no user complaints. A second, unrelated fault was reported at the same time
Trigger A production change that added 16 and destroyed 8 firewall rule collections in one deployment
What happened The canary run had already hit the Terraform time-out. The team went ahead on prod. Azure does not accept concurrent updates of several rule sets, so the updates queued and the apply was aborted by the time-out. Some collections had been created in Azure but were missing from the state. Re-running failed with "already exists". To get the remaining rules in quickly, the team destroyed and recreated existing collections
Contributing factors The 8 destroys ran first, so traffic was already without rules while the adds were retried. A missing private DNS record caused 502 errors at the same time, which made the incident look bigger than the firewall change

Lessons:

  • A time-out on canary is a stop signal. Revert, split the change into smaller pull requests, and deploy those one at a time.
  • When a rule is replaced by a new one, deploy the new rule first and remove the old one in a later change.
  • On "already exists" errors, import the resource into the state (terraform import). Do not destroy production resources to make the state match. Read the plan first (Firewall rules).
  • Tell the customer before the change that short interruptions are possible, and give updates during the deployment.
  • Check whether two symptoms really share a cause (Incident management).

A site-to-site VPN connection showed connected but passed no traffic
Impact Critical: no working connection between production workloads on-premises and in Azure
Trigger Unknown. Logs on the Azure side showed nothing. The state of the connection was connected
Fix A major incident call with the on-premises network partner. The connection was reset on both sides (edit the link and save without changes). The first reset did not help; a second one on the Azure side restored traffic a few minutes later
Root cause Not confirmed. Assumed to be a short disturbance on one side that interrupted traffic and did not restart the tunnel properly

Lessons:

  • Run two connections where possible, so that one fault is not an outage.
  • Alert on traffic flowing, not only on a change of state. Add this to your monitoring rules.
  • Decide in advance who may reset which side, and in which order.

More lessons, in short

  • Naming that includes the environment twice leads to wrong deletions. On a managed platform a configuration set was named after an environment, so the VM it created had that environment twice in its name. The wrong set was removed and a data disk was destroyed. Keep environment names out of configuration set names, compare the plan with the actual resources before you apply a destroy, and ask the owner to confirm the exact resource before you delete it. Four-eyes review only works when the second person checks the actual resources, not only the code.
  • A destroy that failed halfway leaves a half-built resource. If a destroy fails, for example because a VM was stopped and its extensions could not be removed, check what is still there and what the re-run will recreate. A recreated data disk is empty and not attached.
  • Problems found weeks later are expensive. Verify the result of a fix against the real service (mounts, data, traffic), not only against a green pipeline.