AKS Platform: Operations¶
This page is for the platform team (XMMS in the managed model, your own team when self-managed, see Operating models). It describes the recurring work: upgrades, start and stop of non-production, monitoring and alerting, break-glass access and the known gaps.
Reference design, your deployment may differ
Schedules and thresholds are the values of the reference configuration. Support hours, response times and escalation are set by your contract or your own process, see Operating responsibilities and Incident management.
Kubernetes and node image upgrades¶
Upgrades follow the maintenance windows in Cluster baseline.
| Step | Action |
|---|---|
| 1 | A new AKS minor version becomes available. The design wants a supported GA version that is at least one minor version behind the latest. |
| 2 | Check for removed Kubernetes APIs in the platform and application manifests (the design names kubent and pluto). |
| 3 | Upgrade non-production: change aks_kubernetes_version in the non-production configuration and run the pipeline. Patch versions on the patch channel arrive on their own. |
| 4 | Validate platform add-ons and workloads. The design asks for a validation period of about two weeks before production, with automated tests. |
| 5 | Upgrade production in its window with the same change on the production configuration. |
| 6 | Node image upgrades follow the node OS window. They are non-disruptive when workloads have disruption budgets, and with the 33% surge setting. |
A minor version cannot be rolled back. Do not change the version in the portal, because Terraform reports it as drift.
Start and stop of non-production¶
A scheduled workflow stops the non-production cluster outside office hours and starts it again, to save cost.
| Setting | Value |
|---|---|
| Schedule | Monday to Friday. Start at 07:00 UTC (09:00 CET), stop at 15:00 UTC (17:00 CET) |
| Behaviour | Reads the power state first. Starts only a stopped cluster, stops only a running cluster, skips otherwise |
| Manual | The workflow can also be run by hand with start or stop |
| Command | az aks start and az aks stop without waiting |
| Cluster | The cluster name and resource group are set in the workflow for the standalone non-production cluster |
Check how the schedule meets the maintenance window
The schedule is in UTC, so local times shift by one hour with daylight saving time. The non-production maintenance windows start on Tuesday 20:00 (UTC+1), which is after the stop time. A stopped cluster is not upgraded, so the window is missed unless the cluster is running. Confirm how your platform handles this, for example by moving the window or by skipping the stop on that day. The workflow does not cover the hub-integrated non-production cluster.
Monitoring and alerting¶
The Azure resources of the solution switch on monitoring only when you provide a Log Analytics workspace. The in-cluster monitoring stack is deployed by the platform GitOps repository. The load testing design describes the reference non-production cluster as follows.
| Component | Detail |
|---|---|
| Metrics | kube-prometheus-stack in the monitoring namespace, with kube-state-metrics and node-exporter. Retention 7 days or 8 GB |
| Dashboards | Grafana from the same stack. Dashboards are provisioned from config maps with a label |
| Logs | Grafana Alloy as a daemon set, shipping to Loki. Loki is a data source in Grafana |
| Alert rules | Prometheus rules in the GitOps repository: SLO burn-rate alerts and recording rules |
| Routing | Alertmanager groups by alert name, namespace and SLO. Critical alerts go to PagerDuty (repeat every hour), warnings go to PagerDuty (repeat every 4 hours), the Watchdog alert goes nowhere |
| Incidents | PagerDuty service per environment, from the on-call Terraform (Deployment) |
The design defines these service level objectives and alerts them with a fast and a slow burn rate (14.4 times and 3 times the error budget):
| Area | Objective |
|---|---|
| Kubernetes API server availability | 99.9% |
| Ingress success rate | 99.9%, with p95 latency below 500 ms and p99 below 1 s |
| Workload pod availability | 99.5% |
| DNS | 99.99% success |
| GitOps sync | 100% of applications synced |
Other alerts in the design: pod restart rate, crash loops, deployments without ready replicas, certificates, and the security alerts in Identity and access. To check the whole chain from rule to incident, use the load test. See also the alert catalogue for the alerts of the MCP platform itself.
Break-glass activation¶
- The engineer activates the break-glass group in PIM with a justification. Activation is time-limited (4 hours at most) and needs MFA.
- The activation raises a critical alert, so the platform team is aware immediately. The Kubernetes audit log records what the group does.
- Argo CD reverts changes made by hand, so bring any change you want to keep into Git.
- The group membership is reviewed every quarter.
Rebuild a cluster¶
The cluster is designed to be rebuilt: run the pipeline for the environment, then let GitOps restore platform services, RBAC, policies and applications. Persistent volumes are not restored by this. Production data needs its own backup. Resource locks and state protection are described in Cluster baseline.
Policy exception review¶
Review all active policy exceptions every quarter. Remove expired ones. See Security policy.
Known issues¶
| Issue | Workaround | Status |
|---|---|---|
Graph API problem breaks apply with target all when cluster role assignments change |
Use the cicd-aks-rbac target (Deployment) |
Waiting for a provider fix |
| Hub-integrated mode: container registry private DNS zone is required but no registry private endpoint is created | The hub provides one, or add it to this repository | Open item in the design |
| Route table ownership in the hub-integrated mode is not finalised | Decide per environment | Open item in the design |
Azure Local: Arc Resource Bridge cannot reach the cluster control plane (APIServerNotOnline) |
Open TCP 6443 between the management network and the logical network (Prerequisites) | Customer network action |