AKS Platform: Deployment

This page describes how the solution is deployed: the Terraform layout, the pipelines, the configuration files, the roles of the pipeline identity and the order of a first deployment. The pipelines are run by the platform team (XMMS in the managed model, your own team when self-managed, see Operating models).

Reference design, your deployment may differ

Pipeline names, inputs and environment names are those of the reference repository. Your platform may run the same Terraform from another pipeline.

Terraform layout

terraform/
  environments/
    azure/            # root for AKS in Azure
      config/         # <environment>.tfvars and backend-<environment>.hcl
    azure-local/      # root for AKS on Azure Local, with scripts/bootstrap-aks-arc.ps1
  modules/
    azure/            # composition per layer for Azure
    azure-local/      # composition per layer for Azure Local
    network/ identity/ secrets/ cluster-baseline/ security-policy/   # building blocks
  pagerduty/          # on-call setup, separate root and state

Item	Value
Terraform version	1.5 or higher (~> 1.5 in the pipeline)
Providers	azurerm 4.x, azuread 3.x (the Azure root adds an aliased azurerm provider for the hub subscription)
State	Azure Storage, one backend file and state key per platform and environment
Working directory	terraform/environments/<platform>/
Variables	-var-file=config/<environment>.tfvars

Manual run, from the working directory:

terraform init -backend-config=config/backend-<environment>.hcl
terraform plan -var-file=config/<environment>.tfvars -out=tfplan
terraform apply tfplan

Pipelines

Pipeline	Platform	Inputs
Terraform Platform Pipeline (GitHub Actions)	azure and azure-local	platform, environment (nonprod, prod, nonprod-integrated, prod-integrated), action, target_module, vars_ref
Terraform platform pipeline (Azure DevOps)	azure and azure-local; the sources describe it as used for Azure Local	platform, environment (nonprod, prod), action, target_module
Terraform PagerDuty pipeline (GitHub Actions)	On-call setup	environment, action
AKS non-production start and stop (GitHub Actions)	Cost saving	action (scheduled and manual)
Load test (GitHub Actions)	Non-production	Rates, durations, failure trigger (Load testing)

Actions

Action	Effect
plan	Terraform plan only
apply	Plan, then apply
destroy-plan, destroy	Plan the destruction, or destroy
bootstrap	Azure Local only: create the cluster and MetalLB
bootstrap-and-apply	Azure Local only: bootstrap, then plan and apply
bootstrap-and-plan	Azure Local only, Azure DevOps pipeline: bootstrap, then plan

Target layers

target_module is all for normal operation. A single layer (networking, identity, security-policy, cluster-baseline, secrets) uses Terraform -target and is meant for bootstrap or controlled recovery only. The pipeline warns when you use it. The GitHub pipeline adds cicd-aks-rbac, which targets only the pipeline role assignments on the cluster.

What a run does

1. Resolves the GitHub Environment from platform and environment. The -integrated environments reuse the GitHub Environments of the standalone clusters; no new environments are created.
2. Reads the authentication values (auth.conf) and, for Azure Local, the bootstrap values (bootstrap.conf) of that environment from a separate private variables repository, with a GitHub App token.
3. Logs in to Azure with GitHub OIDC (no client secret) and sets the Terraform ARM_* variables.
4. Runs terraform fmt -check, init, validate, then plan or apply for the chosen layer.
5. On Azure Local, runs the bootstrap first when the action asks for it and passes cluster_id and oidc_issuer_url to Terraform through a generated config/ci.auto.tfvars.

One run per platform and environment at a time (a concurrency group). Production approvals are enforced with the GitHub Environment protection rules, or with approval gates on the Azure DevOps environment.

The Terraform state storage account of the hub-integrated environment is in the subscription of the standalone clusters, while the resources are created in the spoke subscription. The backend file therefore pins the state subscription.

Pipeline identity roles

The identity of the Azure pipeline needs, to deploy the full platform:

Role	Scope	Purpose
Contributor	Subscription	Create and manage the resources
Resource Policy Contributor	Subscription	Create Azure Policy assignments
Role Based Access Control Administrator	Subscription, with a condition	Create role assignments for the managed identities
Storage Blob Data Contributor	State storage account	Read and write the state
Graph permission Group.ReadWrite.All (admin consent) or the Entra role Groups Administrator	Tenant	Create and manage Entra ID groups

Restrict the Role Based Access Control Administrator role with the condition "allow user to only assign selected roles to selected principals", and allow only: Managed Identity Operator (cluster identity to kubelet identity), Network Contributor (cluster identity on the VNet and subnets), Private DNS Zone Contributor (cluster identity on DNS zones), AcrPull (kubelet identity on the registry).

More roles in the code

The code also assigns Key Vault Secrets User to workload identities and Azure Kubernetes Service RBAC Admin to pipeline identities. Add these roles to the condition if the pipeline identity creates those assignments. In the hub-integrated mode the identity also needs access in the spoke subscription (Networking).

First deployment of an environment

1. Prepare the configuration file and the backend file for the environment, and the prerequisites (Prerequisites).
2. Run plan with target_module = all and review it.
3. Run apply. The layers are applied in dependency order: networking and identity first, then security policy, cluster baseline and secrets.
4. Read the cluster's OIDC issuer URL, set oidc_issuer_url in the configuration, and apply again. Only then are the workload identities and their federated credentials created.
5. Set the Entra ID group memberships, then let GitOps apply namespaces, RBAC, policies and platform services.

For Azure Local, run the bootstrap first (AKS on Azure Local).

Known issue: Graph API

A Microsoft Graph API problem breaks terraform apply with target all when role assignments on the cluster change (a provider issue in the azuread provider that was waiting for a fix). The workaround is to change the pipeline role assignments through the cicd-aks-rbac target. The role assignments for the pipeline sit outside the identity module for that reason.

On-call setup (PagerDuty)

A separate Terraform root terraform/pagerduty has its own state and pipeline. It creates, per environment:

Resource	Setting
Team and memberships	Users given in the configuration, with a role
Schedule	Weekly rotation in the Amsterdam time zone, restricted to office days and hours
Escalation policy	One schedule, escalation delay 30 minutes, repeated twice
Service	Creates alerts and incidents, auto-resolve after 4 hours, acknowledgement timeout 30 minutes, urgency by severity in support hours and low outside them
Alert grouping	Intelligent grouping
Service dependency	The service supports a business service of the provider
Integration	Events API v2 integration for Prometheus Alertmanager

The hours, escalation steps and who is on call are agreed in your contract or your internal on-call process. The values in the code are the defaults.

Related

• Prerequisites
• Operations
• Architecture
• Platform deploy flow