Skip to content

Roll Out Monitoring

Use this runbook when you set up alert routing for a platform for the first time, when you move it to a different on-call tool, or when you take over a platform (Day-one checklist). At the end, alerts of severity 0 to 2 reach the people who handle incidents, and the alert policies are assigned.

Background is in How alerting works. Examples use an on-call tool that opens an incident from a webhook and posts it in a chat channel. Any tool with a webhook receiver works.

Before you start

Decision Options
Who receives alerts The platform team (XMMS or your own team)
When Business hours only, or also outside office hours (24/7). 24/7 is a contract matter in the managed model
Which tool Your incident or on-call tool. It must accept a webhook and, for critical alerts, be able to call or text a person
Which chat channel One channel per platform or customer for the incident feed
Receivers per environment Canary can send to a test channel. Prod sends to the real route

Make the change through the normal change workflow: branch, pull request, canary, prod.

1. Create the service in the on-call tool

  1. If your organisation groups services by customer or platform, create that grouping first.
  2. Create a service for the platform. Choose an escalation policy that matches the hours you agreed (for example a business-hours policy, or a 24/7 policy).
  3. On the Integrations tab, add an integration for Microsoft Azure (or a generic webhook, if the tool has no Azure integration).
  4. Copy the integration URL. Treat it as a secret: anyone who has it can create incidents.

2. Connect the chat channel (optional)

  1. Create a channel for the platform alerts, internal to the platform team.
  2. In the on-call tool, open the service and add an extension or connection for your chat tool.
  3. Connect the service to the channel. Check that a test event arrives there.

3. Put the receiver in the configuration

Set the receivers of the platform action groups in <primary region>/foundation.auto.tfvars of each platform environment folder. The general action group takes its webhook from monitor_action_group_general_webhook_receiver. The action group that receives the landing zone alerts has its own variables (monitor_action_group_xpirit_*). Set both, or you route only part of the alerts (Hub config).

monitor_action_group_general_webhook_receiver = {
  "Webhook Receiver - <platform name> Hub (Production)" = {
    name                    = "Webhook Receiver - <platform name> Hub (Production)"
    service_uri             = "https://<on-call tool>/integration/<integration-key>/enqueue"
    use_common_alert_schema = true
    aad_auth                = null
  }
}
Field Value
Map key and name A readable name. Use one per environment and say which is canary and which is prod
service_uri The integration URL from step 1
use_common_alert_schema true. The on-call tool receives the common alert schema
aad_auth null unless your receiver validates an Entra ID token

Do not commit a secret URL to a repository that many people can read if your policy forbids it. Check where your configuration repository stores such values and who can read them.

E-mail receivers are set the same way (monitor_action_group_general_email_receiver and the matching variable of the second group). Use a group mailbox, not a person.

4. Apply

  1. Open a pull request and review the plan. The foundation layer should show changes only to the action groups.
  2. Apply to canary, check the action groups in the portal (Monitor > Alerts > Action groups), then apply to prod.

5. Assign the alert initiatives

Check alerting_management_groups in <primary region>/postdeployment.auto.tfvars: the keys cloudplatform and landingzones are required, and each deploy_alerting_* flag switches one initiative on. See Monitoring rules.

Run hub. For existing subscriptions, start remediation tasks so that the rules are created for existing resources (assign the standard rules).

6. Optional: the hub 24/7 route

If after-hours alerts for the hub are in your service, enable hub_247_support with its own webhook (and a different integration in the on-call tool, with its own escalation policy). Settings and defaults: Monitoring rules.

Verify

  1. Rules exist. In the portal, Monitor > Alerts > Alert rules, filtered on a platform subscription.
  2. Policy is compliant. Policy > Compliance for the alert initiatives. Remediate what is not.
  3. A message arrives. Trigger a test event in the on-call tool, or wait for the next Service Health or Resource Health alert, and check that the incident appears, that it posts in the chat channel, and that a critical one reaches the on-call person by the agreed means (call and SMS).
  4. Escalation works. Check that an unacknowledged critical incident escalates as agreed.
  5. Record it. Write down the receivers, the hours, and who owns the integration, in the place where your team keeps its platform documentation.

Operate

  • Review alerts weekly, tune thresholds that are noisy, and check receivers when people join or leave (Keep the alerting healthy).
  • Rotate the integration URL if it leaks, and update the configuration in the same way as above.
  • Keep a runbook for every severity 0 alert (Alert catalogue).

Roll back

Remove or change the receiver in the tfvars file and run hub again. The alert rules stay, but notifications stop, so tell the people who rely on them first.