How Alerting Works¶

The platform creates its alerts with Azure Policy and Terraform, not by hand. This page explains the parts and how an alert travels from a resource to a person. For the list of alerts, see the Alert catalogue. For the steps to connect your own on-call tool, see Roll out monitoring.

The parts¶

Part	What it is	Created by
Alert initiatives	Azure Policy initiatives that contain the alert definitions. They follow the Azure Monitor Baseline Alerts (AMBA) best practice, which Microsoft collected from field experience. Five exist: `Alerting-ServiceHealth`, `Alerting-Management`, `Alerting-Connectivity`, `Alerting-Identity` and `Alerting-LandingZone`	Definitions in the `foundation` layer
Assignments	One assignment per management group, switched on by `alerting_management_groups`	The `postdeployment` layer
Activity log alerts	Alerts on subscription events of the platform subscriptions, enabled by `enable_activity_log_alerts` (default on)	The `foundation` layer
Action groups	The receivers: e-mail addresses and webhooks. Two per region: a general one (`ag-<company>-mgmtgeneral-<env>-<region>-001`) and one for the alerts of the landing zone assignments (`ag-<company>-mgmtxpirit-<env>-<region>-001`)	The `foundation` layer
Alert processing rules	Rules that decide which alerts go to which action group, based on severity	The `connectivity`, `identity`, `management` layers, and the landing zone initiative in each landing zone subscription
Hub 24/7 route (optional)	A dedicated action group and processing rule for after-hours alerts on selected hub components	The `connectivity` layer, switched on with `hub_247_support`

How the assignments are switched on, where the spoke gets its rules, and how to remove them again are described in Monitoring rules. Names of the action groups are in the naming reference.

Why policy¶

Alerts that are deployed by policy follow the resources:

When a resource is created or changed, the deployIfNotExists policy creates the alert rule. Nobody has to remember.
A rule cannot be silently missing: the compliance of the alert policies is visible in Azure Policy and is part of the monthly health check.
For resources that already exist when an assignment is switched on, the rules appear only after a remediation task (Monitoring rules, step 4).

Severity¶

Azure Monitor uses five severities. Differentiating them prevents alert fatigue: if the team is paged for things that are not problems, people start to ignore the messages, and one day they ignore the one that matters. A critical alert needs a fast reaction. An informational alert can wait in the backlog.

Severity	Name	Meaning	What happens
0	Critical	The service is failing or security is at risk. Act now.	Processed and sent to the action group. Call the on-call person if your service level says so
1	Error	Significant degradation	Processed and sent
2	Warning	A performance or capacity threshold is crossed	Processed and sent
3	Informational	Operational awareness	Logged in Azure Monitor. No notification
4	Verbose	Audit trail: administrative and security events	Logged in Azure Monitor. No notification

Severities 0 to 2 are routed by the alert processing rules. Severities 3 and 4 are kept in Azure Monitor for history and compliance reporting. All alerts are logged in Azure Monitor, whatever their routing.

How you handle each level is your decision

The table shows the default routing. Response times for severity 0 to 2, and whether they apply outside office hours, belong in your service levels (Service level management). In the managed model they come from the contract.

How an alert travels¶

flowchart LR
    r["Resource or activity log"] --> a["Alert rule<br/>(created by policy)"]
    a --> p["Alert processing rule<br/>(severity 0 to 2)"]
    p --> g["Action group<br/>(e-mail, webhook)"]
    g --> t["On-call / incident tool"]
    t --> c["Chat channel"]
    t --> o["On-call engineer<br/>(call, SMS)"]

An alert rule fires on a metric or an activity log event.
An alert processing rule selects alerts by severity and sends them to an action group.
The action group calls a webhook at your incident or on-call tool, and optionally sends e-mail.
The tool opens or updates an incident, posts it in the chat channel of the platform team, and, for critical alerts, notifies the on-call engineer by phone call and SMS, if you configured that.
The incident process starts (Incident management).

Keep all alerts of the platform going through the same action groups. A central route is easier to test, to change and to audit than receivers that are spread over many rules.

Optional 24/7 route for the hub¶

Outside office hours you may want only a few hub alerts to wake somebody. The optional hub 24/7 route (ADR-0007) does that, for the hub only. Enabled per platform, it creates its own action group and processing rule on the connectivity subscription, with a webhook that you choose and a daily time window (default 18:00 to 08:00, Western European time). By default the rule covers resource health and service health alerts for the firewall, Virtual WAN, virtual hub, VPN gateway and Application Gateway.

Why a dedicated action group: the webhook of one customer stays separate from the shared channels, and enabling it does not change how other alerts are routed. The cost is one extra action group and a list of alert names that you maintain if the names change upstream. The settings are in Monitoring rules.

A managed service can offer 24/7 handling with an on-call rotation and runbooks for every critical alert. Whether it applies to you is a contract matter.

Spokes and workload teams¶

Subscriptions that sit under the landingzones management group get the landing zone alert rules, and their severity 0 to 2 alerts go to the platform's action group. A workload team can ask for the standard rules to be removed from its spoke, with the MonitorDisable tag or a policy exemption (Monitoring rules).

The platform does not alert on what the application does. The workload team creates those alerts, with its own action groups and on its own workspace (Operate your workload).

Keep the alerting healthy¶

Task	How often	Where
Review the alerts of the week: tune noisy rules, create work items for real problems	Weekly	Maintenance calendar
Check that the alert policies are compliant and the alerts fired in the month were actionable	Monthly	Monthly health check
Check the receivers: people left, a webhook was rotated, the on-call tool changed	Quarterly, and at every team change	`foundation.auto.tfvars` receivers (Roll out monitoring)
Check that a critical alert really reaches a person	After each change to the route	Roll out monitoring

A site-to-site connection that reports connected but passes no traffic is not caught by a state alert. Alert on traffic as well (Known issues).

Upgrades and old alert objects¶

Older versions deployed custom alert policy objects that had to be removed by hand when the alert policies were replaced. Look for them if you upgrade across v4.5.1 or v4.11.x: Platform upgrade guides.