Skip to content

Incident and Major Incident Handling

This page describes how the platform team handles incidents on the platform: the hub, the pipelines, and the baseline of the spokes. It also explains how to raise a priority 1 (P1) incident with the cloud provider's support when the cause is Azure itself.

Who the platform team is depends on your operating model. Severities, response times, hours and contact details come from your contract (managed service) or from your own definition (self-managed). They are written here as placeholders. For what workload teams do and what to include in a message, see Support. For a first severity model, see Operating responsibilities.

Incident, service request or change

It is… When Handled as
Incident Something that worked no longer works, or works worse This page
Service request A workload team asks for something the platform offers How to request, a standard change (Common requests)
Change The platform team changes the platform (code version, configuration, a setting in Azure) Change workflow

Keep them apart in your ticket system. Two reasons: resolve times mean something different for each, and many incidents are caused by a change that nobody recorded (see the first lesson in Post-mortems).

Handling an incident

flowchart LR
    a["1. Register"] --> b["2. Acknowledge + classify"] --> c["3. Diagnose"] --> d["4. Restore service"]
    d --> e["5. Close + inform"] --> f["6. Post-mortem (sev 1 / major)"]
  1. Register. Every incident gets one record in the agreed ticket system, even when it was reported by chat or phone. Minimum fields: reporter, workload and spoke (or hub component), description, time it started, whether production is affected, screenshots or error messages. A P1 that is only handled by chat leaves no trail for the post-mortem.
  2. Acknowledge and classify. Reply within the response target and set the severity. Re-classify when you learn more. A problem that looks small can be severity 1 if production has been down since the weekend: classify on impact, not on how long it has been going on.
  3. Diagnose. Start with the checks in Where to look first. Write findings in the record as you go.
  4. Restore service. The first goal is to restore the service, the second is to find the cause. A temporary fix is fine if you record it and plan the permanent one. Do not fix an incident by changing Azure by hand without recording it: the next pipeline run reverts it, or it hides the real cause (drift).
  5. Close and inform. Tell the reporter what was wrong and what you did. Close the record with the cause and the fix.
  6. Post-mortem. Required for every severity 1 incident and every major incident (Post-mortems).

Severity

Use the severity model of your contract or your own. A model that has been used on managed platforms is in Operating responsibilities. Whatever the numbers, write down for each severity:

Item Value
Definition, with one or two examples from your platform <definition>
Hours in which it is handled <hours>. Say clearly whether severity 1 is handled outside office hours, and for which components
Time to respond <response target>
Update frequency to the reporter <every n minutes / hours>
Who is called, and how <on-call contact>

Major incidents

A major incident is an incident with severe business impact that needs more than the normal flow: several people at the same time, a shared view of the situation, and a communication lead. Typical triggers:

  • production applications of one or more workload teams are unreachable;
  • the connection between Azure and on-premises is down;
  • a platform change went wrong in production and cannot be reverted quickly;
  • an incident is severity 1 and the first diagnosis did not find the cause within <n minutes>.

Roles

For a small platform team one person can hold more than one role, but name them out loud at the start.

Role Does
Incident manager Owns the incident until it is closed. Decides on severity, escalation and when to stop. Does not do the technical work
Technical lead Leads diagnosis and restoration. Decides which change is tried next
Communication lead Keeps stakeholders informed at the agreed frequency. Single point of contact for the customer, workload teams and partners
Scribe Writes down the timeline with time stamps: what was seen, what was decided, what was changed, by whom
Partner and provider contacts Engineers from other parties involved, for example the on-premises network or firewall team, the application owner, the cloud provider's support

How it runs

  1. Open a bridge (call or chat room) as soon as the incident is declared major, with everyone who can change something: the platform team, the owner of the affected workload, and the owners of connected systems (on-premises network, identity). Record the bridge location in the ticket.
  2. Agree the facts: what is affected, since when, and what changed recently (platform changes, releases, other teams' changes).
  3. Stop other changes to the affected area until the incident is closed.
  4. Work in short cycles: hypothesis, check, act, record. Change one thing at a time, and write it down.
  5. Communicate on a fixed rhythm, also when there is nothing new. Each update says: current impact, what is known, what is being done, next update time.
  6. Declare resolved when the service is restored and the reporter confirms it. If a workaround is in place, keep the problem open.
  7. Plan the post-mortem within <n> working days (the severity model of the managed service uses five).

What worked in real incidents

  • A named person who gave a regular overview of the status, so that everybody else could work.
  • People who joined the call at short notice, and knowledge of the platform that was good enough to find the cause fast.
  • Involving the other parties (on-premises network team, partner) in the same call instead of passing messages around.

Where to look first

Most incidents on the platform fall in a few groups. The table is a start for the diagnosis, not a complete runbook.

Symptom Look at Typical causes seen
Users get 403 Forbidden on an application behind the Application Gateway WAF logs: which rule matched, for which request A WAF setting or rule set was changed without testing (for example inspection of the request body). Disable the matching rule temporarily, then fix the cause. See Application Gateway
Users get 502 Bad Gateway Backend health of the Application Gateway; DNS resolution of the backend name from the hub; firewall logs The backend name cannot be resolved because the private DNS record is missing, so traffic never reaches the firewall. A firewall rule that was just recreated can look like the cause but is not always
A workload cannot reach a destination Firewall logs for the source and destination (network rule and application rule tables, see below); NSG and route table of the spoke Missing or removed rule, wrong IP group, rule collection in the middle of a deployment (Firewall rules)
Connection between Azure and on-premises is down, and the portal says connected Traffic on the connection, not only its state; the other party's logs; whether a second connection works The connection state is connected but no traffic flows. Resetting the connection on both sides (edit the link and save without changes) restored traffic in a reported case. Agree who resets which side before you start
A pipeline run fails or times out The failing job and layer; Known issues Time-outs on large firewall changes, a state that differs from Azure, an expired deployment secret
Many alerts at once Azure Service Health and Resource Health A provider-side problem. See Raise a P1 with the cloud provider

Do not assume two symptoms share one cause

During one reported incident, a firewall rule change and a missing DNS record produced the same 502 at the same time. Test each cause separately before you conclude.

Logs

Platform logs are normally sent to a central Log Analytics workspace and are queried with Kusto (KQL).

Source Where Notes
Azure Firewall The firewall resource, Logs. Resource-specific tables such as AZFWNetworkRule, AZFWApplicationRule, AZFWNatRule, AZFWDnsQuery, AZFWFlowTrace and AZFWIdpsSignature (and the aggregation variants) Needs firewall diagnostics enabled to Log Analytics (Firewall rules)
Application Gateway and WAF The Application Gateway resource, Logs Depending on the diagnostic setting the data is in the shared AzureDiagnostics table (the older mode) or in resource-specific tables
Workload logs The spoke's own Log Analytics workspace Owned by the workload team (Operate your workload)

Access to the logs is a role on the workspace tables and Monitoring Reader on the resource. On a managed platform, give workload teams a read-only group for their own traffic when they need to troubleshoot it themselves, instead of asking the platform team to read logs for them.

Raise a P1 with the cloud provider

Some incidents are caused by Azure, not by the platform or by the workload. The platform team (or the workload team, if your setup lets it) then raises a P1 with the cloud provider's support. Who may do this, and through which channel, depends on your support agreement: a Microsoft support plan of your own, or a partner (for example a Cloud Solution Provider) that passes it on. See Operating models and Support.

Is it a provider P1?

Raise a provider P1 for Do not raise one for
An Azure service outage that affects production Performance problems caused by too few resources (scale up, check quota)
An unexpected Azure system failure Problems in the application
Critical availability problems of an Azure database service Resources that are not Azure resources
A disruption of an Azure cloud service Problems in third-party Marketplace applications
Subscriptions that are outside the support agreement

Check Azure Service Health and Resource Health first. If Azure already reports the problem, you add your impact to it.

Who raises it

Limit the right to raise provider P1s to a few named people or roles, for example the platform team's engineers on call (<roles allowed to raise a P1>). A workload team that is not allowed to raise one contacts the platform team and says that the problem looks like a provider issue. Write the list in your onboarding document.

What to put in the record

Create an incident with priority critical (P1) in the ticket system or the provider's portal, with a clear title. Fill in the description in this order:

Section Content
Incident overview What: which Azure service is broken. Impact: number of users or services. When: since when. Status: the current situation. Errors: the relevant error messages
Technical details Resource IDs of the affected resources, the subscription, related monitoring alerts, relevant log lines
Contact information Primary contact with a phone number, backup contact, the channel where updates are posted (<update channel>), other stakeholders to inform
Reproduction steps Exact steps to see the issue: where to look (portal path, resource name), what to do, what should happen, what happens instead

During the incident:

  1. Keep the description up to date.
  2. Put a time stamp on every update.
  3. Record every temporary fix and every attempted solution.
  4. Tell the affected workload teams where they can follow the status.

Contract-dependent

Whether provider P1s are handled 24 hours a day, and by whom, is part of your contract or support plan. Do not promise round-the-clock handling to workload teams unless it is agreed for the hub (Support).