Monitoring and Alerting¶
Nobody wants to watch dashboards all day and wait for something to break. The platform therefore deploys its alerting as code and enforces it with Azure Policy, so the alerts that should exist do exist, and a notification reaches the people who can act on it.
This section explains how alerting works, which alerts exist, and how to set up (or take over) the routing of alerts.
Who receives and handles alerts depends on your operating model. In the managed model the receiver is the XMMS on-call route. In the self-managed model it is your own platform team.
Pages in this section¶
| Page | Use it to |
|---|---|
| How alerting works | Understand the alert initiatives, severities, action groups, processing rules and the optional 24/7 route |
| Alert catalogue | See what is monitored, by severity and by resource area |
| Roll out monitoring | Connect the platform to your on-call tool and turn the alerting on |
| Monitoring rules (assign or remove for a spoke) | Change the standard alert rules of one spoke |
| Monthly health check | Check that the alerts and their policies are in place and not too noisy |
Monitoring and observability¶
| Term | Meaning | In the platform |
|---|---|---|
| Monitoring | Collecting known indicators (availability, CPU, throughput) and acting on thresholds | Metric and activity log alerts from the alerting initiatives |
| Observability | Being able to ask new questions of the system, including questions you did not plan for | Logs and metrics in Log Analytics, searchable and correlatable. A dashboard tool on top (for example Grafana) gives one place to start and to drill down into the portal |
Alerts tell you that something you expected has happened. Logs let you find out what you did not expect. Use both. For a first diagnosis, see Where to look first.
Who is responsible for what¶
| Area | Responsible | Notes |
|---|---|---|
| Hub alerts, platform action groups, routing | Platform team | The foundation, connectivity and postdeployment layers (Deployment layers) |
| Standard alert rules in the spoke subscriptions | Platform team deploys them by policy | The workload team can ask to remove them (Monitoring rules) |
| Alerts for the application itself | Workload team | Alert rules, action groups and dashboards on the workload's own workspace (Operate your workload) |
| The people and the tool that receive alerts | Whoever runs the service (Operating models) | Contact lists and escalation are placeholders in your contract or your own policy |
The principle behind it: the team that builds a workload is responsible for its whole lifecycle, including its alerting. The platform provides a baseline so that teams do not start from nothing.
Measuring the service: example objectives¶
To judge whether the platform is healthy, define a few objectives that are measurable and lead to action. Measure them the same way every month and report them (Service level management).
| Objective | Description | Example target | Measured with |
|---|---|---|---|
| Time to recovery | Mean time to recovery of severity 1 incidents | 1 hour, in business hours | Time from detection to resolution in the ticket system |
| Availability | Percentage of time the platform is available | The Microsoft SLA of the components | Service Health |
| Deployment success rate | Share of successful production deployments | 99% | Pipeline success rate |
These are examples from a managed platform, not defaults. Choose your own targets and say where the numbers come from. A dashboard that reports them (for example in Grafana) saves time at the monthly review.