AKS Platform: AKS on Azure Local¶
This page describes the second variant of the solution: AKS on Azure Local (also called AKS Arc). The Kubernetes cluster runs on Azure Local hardware in a data centre and is managed through Azure Arc. The management plane is in Azure, the runtime plane is on premises.
Less complete than the Azure variant
The sources document the design, the bootstrap script and the pipelines for this variant. In the committed configuration
files every layer switch (deploy_identity, deploy_cluster_baseline, deploy_arc_networking, deploy_argocd,
deploy_azure_policy_extension, deploy_security_policy, deploy_key_vault) is false. This documentation cannot say
which layers are active in a live environment. Ask your platform team.
Differences from the Azure variant¶
| Aspect | AKS in Azure | AKS on Azure Local |
|---|---|---|
| Management plane | Azure | Azure and Azure Arc |
| Runtime plane | Azure | Azure Local, on premises |
| Cluster resource | Terraform (azurerm_kubernetes_cluster) |
Azure CLI (az aksarc create) in a bootstrap step |
| Networking | VNet, subnets, NSG | A logical network that already exists. Terraform only passes its ID through |
| Control plane | Azure-managed | Runs on Azure Local hardware |
| Load balancing | Azure load balancer | MetalLB, enabled by the Arc Networking extension and configured with az k8s-runtime |
| GitOps | Argo CD from the GitOps repository | Argo CD as an Arc extension (microsoft.argocd) |
| Policy | Azure Policy add-on | Azure Policy extension for Arc (microsoft.policyinsights) |
| Workload identity | AKS built-in | Enabled at cluster creation (--enable-oidc-issuer --enable-workload-identity) |
| Identity and secrets | Entra ID, managed identities, Key Vault | The same shared modules |
| Defender for Containers | Optional | Not supported on Azure Local |
| AKS managed add-ons | Available | Not available, Arc extensions are used instead |
| Pipeline | GitHub Actions | Azure DevOps (customer-provided agent) or GitHub Actions |
What is and is not managed by Terraform¶
| Not managed by Terraform | Reason |
|---|---|
| The Azure Local cluster | Physical infrastructure with its own lifecycle |
| Arc Resource Bridge, custom location, logical network | Prerequisites, must exist and be healthy (Prerequisites) |
| VNets, subnets, NSGs, firewall, NAT gateway | Not applicable, physical or software-defined networking |
| API server VNet integration | Not applicable, the control plane is on premises |
| AKS cluster, OIDC issuer, workload identity | Created by the bootstrap |
| MetalLB load balancer and BGP peers | Created by the bootstrap with az k8s-runtime |
| Managed by Terraform | Resource |
|---|---|
| Arc extensions: Argo CD, Azure Policy, Arc Networking | azurerm_arc_kubernetes_cluster_extension |
| Entra ID groups, break-glass group | azuread_group (shared module) |
| Managed identities and federated credentials | azurerm_user_assigned_identity, azurerm_federated_identity_credential |
| Azure role assignments | azurerm_role_assignment |
| Key Vault | azurerm_key_vault, without a private endpoint in this variant |
| Azure Policy assignment (pod security baseline) | azurerm_resource_group_policy_assignment |
| Resource groups | rg-<env>-identity-arc, rg-<env>-secrets-arc, rg-<env>-security-arc |
Deployment order¶
flowchart TB
pre["Prerequisites:<br/>Azure Local, Arc Resource Bridge,<br/>custom location, logical network,<br/>firewall path to the control plane"] --> boot
boot["Phase 1: bootstrap<br/>(PowerShell and Azure CLI)"] --> tf
tf["Phase 2: Terraform<br/>(extensions, identity, policy, Key Vault)"] --> gitops
gitops["Argo CD applies the GitOps repositories"]Phase 1: bootstrap¶
The bootstrap script validates its inputs, makes sure the Azure CLI extensions aksarc and k8s-runtime are present,
registers the resource providers it needs, and then:
- creates the resource group, if it does not exist;
- creates the AKS Arc cluster with the custom location, the logical network, the Entra ID admin group, SSH key generation and the OIDC issuer and workload identity switched on. If the cluster exists, it skips the creation;
- checks that the OIDC issuer is enabled and reads its URL;
- enables the Arc Networking runtime for MetalLB (with a fallback that installs the
microsoft.arcnetworkingextension); - creates the MetalLB load balancer with the IP range and the advertise mode (
ARP,BGPorBoth); - creates a BGP peer when the mode needs one (peer name, both AS numbers and the peer address are then required);
- writes the cluster ID and the OIDC issuer URL to an output file.
Resource providers registered or checked: Microsoft.Kubernetes, Microsoft.KubernetesConfiguration,
Microsoft.ExtendedLocation, Microsoft.AzureStackHCI, Microsoft.KubernetesRuntime. The design document also lists
Microsoft.HybridConnectivity and Microsoft.ResourceConnector.
Phase 2: Terraform¶
Terraform needs two values from the bootstrap, cluster_id and oidc_issuer_url. The pipeline writes them to
config/ci.auto.tfvars at run time, from the bootstrap output when both ran in the same run, otherwise from pipeline or
environment variables. Then the layers are enabled with the switches, in this order:
| Step | Switch | Result |
|---|---|---|
| 1 | deploy_cluster_baseline |
Container for the extensions |
| 2 | deploy_argocd |
Argo CD extension, release train Stable by default |
| 3 | deploy_azure_policy_extension |
Azure Policy for Arc |
| 4 | deploy_arc_networking |
Arc Networking extension for MetalLB |
| 5 | deploy_identity |
Groups, identities, federated credentials |
| 6 | deploy_security_policy |
Pod security baseline initiative on the cluster resource group, effect Audit |
| 7 | deploy_key_vault |
Key Vault |
Terraform installs only the Argo CD extension. Applications, application sets and projects come from the GitOps repositories.
The expected GitOps layout is one folder per cluster and environment (clusters/azure-local/nonprod, .../prod) in both
the platform and the application repository.
Arc Networking and a service principal
The Arc Networking extension can need the object ID of a Microsoft first-party service principal
(arc_networking_fpa_object_id). The bootstrap tries to resolve it in its fallback path and stops with a message when it
cannot. If your account has no directory read permission, supply the ID in the configuration.
Not available or unclear¶
- Key Vault access from the cluster. The shared Key Vault module disables public network access, and the Azure Local variant creates no private endpoint. A cluster on premises then cannot reach the vault unless the network design adds a private path. The sources do not describe one.
- Kubernetes policy and RBAC from GitOps are described for the Azure variant. The Azure Local pages only name the ArgoCD extension and the GitOps folder layout.
- Node pool sizing, upgrades and maintenance windows of the Azure Local cluster are not set by this solution. The
bootstrap script creates the cluster with the default sizes of
az aksarc create. - Monitoring and backup for Azure Local are not described.