Maintenance Calendar¶
Most platform problems are cheaper to prevent than to fix. This page lists the recurring tasks of the platform team by frequency, and how to announce maintenance. Use it to build a calendar for your team. Frequencies are suggestions: your contract or your own policy decides.
In the managed model the platform team (XMMS) does these tasks under the contract. In the self-managed model it is your own platform team (Operating models).
Recurring tasks¶
| Frequency | Task | Where |
|---|---|---|
| Weekly | Drift check: run hub in Plan mode. Any change that you did not commit is drift |
Operating responsibilities |
| Weekly | Spoke drift: read the latest spoke drift report, if you run the spoke drift pipeline | Spoke drift detection |
| Weekly | Review alerts of the week: tune noisy rules, create work items for real problems | Monitoring rules, How alerting works |
| Weekly | Run the secret watcher, and act on its alerts | Secret watcher |
| Weekly | Check the status of the pipelines of the platform (failed or stuck runs) | Pipelines and workflows |
| Monthly | Monthly health check and report | Monthly health check |
| Daily (automated) | Billing export run, if you report cost centrally. Check a failed run | Billing export |
| Monthly | Review policy exemptions (expires_on) |
Policy assignments |
| Monthly | Review Defender for Cloud and Advisor recommendations | Monthly health check |
| Monthly | Secrets and certificates that expire within the lead time (for example 60 days) | Secret and certificate expiry |
| Per release | Take a new platform version: canary first, then prod. Read every upgrade guide between the current and the target version | What to do when a new version is released, Platform upgrade guides |
| Per release | Check that the code reached your copy of the code repository (new tag and branch history present) | Code synchronisation |
| Per release | Rebuild the container image and the runners | What to do when a new version is released |
| Quarterly | Review the service level, post-mortem actions and the roadmap | Meetings |
| Quarterly | Check alert receivers and the on-call route: people who left, rotated webhooks, a changed tool. Send a test alert | Roll out monitoring |
| Quarterly | Review access: who has which role on the platform subscriptions, and the guest and disabled accounts | Access and PIM |
| Yearly | Review the contact list, on-call route and escalation path for major incidents, and the list of people who may raise a provider P1 | Incident management |
| When needed | Decommission spokes and subscriptions that are no longer used | Delete a spoke |
Secret and certificate expiry¶
Expired secrets and certificates are a frequent cause of failing pipelines and applications.
- List everything that expires within the lead time: deployment identities, spoke service principals, App Gateway certificates in Key Vault, and application registrations that exist in the tenant.
- For each item, find the owner. For platform identities that is the platform team. For application registrations that belong to a workload, the workload team.
- Rotate or renew before expiry, or replace the secret by a federated credential where the service supports it.
- Register the rotation with the change record, and check the next run of the pipeline that uses the secret.
- Remove registrations that nobody owns, after a notice period.
A scheduled check that alerts before expiry is described in Secret watcher. Rotation details for the platform's own identities are in Operating responsibilities and Change spoke.
Announcing maintenance¶
For planned work that can interrupt workloads (a new platform version, firewall changes that may cause short interruptions, patch windows):
| Step | What |
|---|---|
| Plan | Choose a window outside peak hours of the workloads, and avoid known business peaks. Check that no other change is planned for the same area |
| Announce | Send the notice to the <announcement channel> and to the affected workload contacts, with the lead time of your service level: what, when, expected impact, who to contact |
| Test | Run the change on canary first. If canary shows time-outs or unexpected destroys, stop and split the change (Post-mortems) |
| Do | Keep the affected workload contacts informed during the work, at the agreed rhythm |
| Close | Tell them when it is done, and record the result |
Patch windows for managed VMs (only if your platform offers them)¶
The MCP core does not patch the VMs of workload teams: that stays with the workload team (Responsibilities). If your platform adds a managed VM offering, a common pattern is to patch with Azure Update Manager and maintenance configurations, assigned automatically from a tag. A pattern from a managed platform:
| Item | Example |
|---|---|
| Classes | Business-critical and non-business-critical. The class follows from the business classification of the workload |
| Windows | One monthly window per class, outside office hours, about two hours, in a named time zone. The business-critical window is one week after the non-business-critical window, so that problems show up first on the less critical systems |
| Urgent patches | Out-of-band: the non-business-critical class the same day, the business-critical class the day after |
| Content | Critical, security and update-rollup classifications for Windows. Critical, security and other for Linux |
| Reboot | Reboot if required |
| Assignment | A tag on the VM (for example PatchPriority) selects the maintenance configuration through a dynamic scope, so new subscriptions and VMs are picked up without edits to the configuration. Untagged VMs are not patched by it |
| Stopped VMs | A VM with an auto-shutdown schedule is started before the window and stopped afterwards with a pre-event and post-event |
| Notifications | Optional: a message to the workload team after the update, or a few days before |
Allow at least 40 minutes before the start of the window for pre-events to run. If a VM cannot fit in a window, agree an exception with the platform team. Failed patch runs are listed in the monthly health check.