AKS Platform: Networking¶
This page describes the network design of the Azure variant: the spoke, the two network modes, egress, DNS, private endpoints and ingress. The Azure Local variant uses an existing logical network instead (AKS on Azure Local).
Reference design, your deployment may differ
Address ranges and names are the defaults of the reference configuration. Use ranges that do not overlap with your other networks. The design documents describe an Azure Firewall based egress model. Where the Terraform code does less than the design says, this page says so.
Network modes¶
The networking layer has two modes and exposes the same outputs in both, so the layers above it (cluster baseline,
secrets, security policy) do not change. The mode is the variable network_mode; the default is standalone, so existing
deployments keep working.
standalone |
hub_integrated |
|
|---|---|---|
| Spoke VNet and subnets | Created by the solution | Created by the hub and landing zone, looked up by name |
| Resource group | rg-<env>-network, created |
The spoke's resource group, looked up |
| Route table | rt-<env>-aks, created, no routes, associated to the node subnet |
None. Egress follows what the hub propagates (routing intent or firewall route) |
| Private DNS zones | Created in the spoke (default: privatelink.vaultcore.azure.net) and linked to the VNet |
Hub-owned. Their IDs are passed in a zone_ids map. The hub links them to the spoke |
| AKS outbound type | loadBalancer |
userDefinedRouting |
| External ingress public IP | Created (static, Standard SKU) | Not created. Inbound arrives through the hub to an internal load balancer |
| Provider | Default azurerm |
Second azurerm provider (alias hub) for the spoke subscription, which can be another subscription than the state |
The existing standalone environments were moved to the new structure with moved blocks. A plan on the live
non-production state showed no changes.
Mode rules are enforced by output preconditions: spoke_config and hub_dns_config are required in the hub-integrated mode,
and the VNet and subnet CIDRs are required in the standalone mode. Preconditions were chosen over cross-variable validation
to keep Terraform at version 1.5 or higher.
Spoke layout¶
Four subnets exist in both modes (names are passed explicitly in the hub-integrated mode).
| Subnet | Purpose | Reference size | Notes |
|---|---|---|---|
snet-aks-nodes-<env> |
Node VMs | /21 (about 2,000 addresses) |
Sized for the maximum node count plus upgrade surge. NSG allows VNet and Azure load balancer traffic inbound and denies the rest. |
snet-ingress-<env> |
Load balancer frontends of the ingress gateway | /24 |
|
snet-private-endpoints-<env> |
Private endpoint network interfaces | /24 |
Private endpoint network policies are disabled on this subnet. |
snet-aks-api-<env> |
API server VNet integration | /28 |
Delegated to Microsoft.ContainerService/managedClusters. |
Pods do not use VNet addresses (overlay), so subnet sizing depends only on nodes and surge. Subnets cannot be resized without recreating them, so size them generously. The design allows a hub with its own address space for the firewall, Bastion and DNS resolver; those are not created by this solution.
AKS network profile¶
| Setting | Value |
|---|---|
| Network plugin | Azure CNI, overlay mode |
| Network policy | Calico |
| Pod CIDR | A range that is virtual and not routed on the VNet, so it can be the same for both clusters in standalone mode |
| Service CIDR and DNS service IP | A different range per cluster; the DNS IP must lie inside the service CIDR |
| Outbound type | loadBalancer (standalone), userDefinedRouting (hub-integrated) |
The design document says the network policy engine is Azure Network Policy in the cluster baseline chapter and Calico in the networking chapter. The code sets Calico.
Egress¶
Design versus code
The design describes all egress going through Azure Firewall (default route 0.0.0.0/0 to the firewall) with a NAT
gateway on the firewall subnet for scalable SNAT. The repository contains modules for the firewall, NAT gateway and DNS
resolver, but the Azure environment root does not use them. In the standalone mode the cluster uses the AKS load
balancer for outbound traffic and the route table has no routes.
In the hub-integrated mode the cluster uses user-defined routing and the hub decides the path. The code comments describe the hub egress as routing intent that forces traffic to the hub Azure Firewall. The design document describes a default route to the hub firewall's private IP instead, applied to all four subnets, with a variable to extend a hub-owned route table. The final route table ownership is listed as open in the design.
When a firewall controls egress, it must allow at least the AKS required endpoints. The design lists:
| Endpoint | Purpose |
|---|---|
*.hcp.<region>.azmk8s.io |
AKS API server communication |
mcr.microsoft.com, *.data.mcr.microsoft.com |
Microsoft Container Registry |
management.azure.com |
Azure Resource Manager |
login.microsoftonline.com |
Entra ID |
packages.microsoft.com, acs-mirror.azureedge.net |
Node components and binaries |
*.blob.core.windows.net |
Node bootstrapping |
| OS update endpoints (Ubuntu or Azure Linux) | Security updates |
dc.services.visualstudio.com and the Azure Monitor endpoints |
Telemetry, when monitoring is on |
Missing rules make cluster bootstrap fail under user-defined routing. Rules for application teams are per team and per environment; see Firewall rules and IP groups for how the MCP hub does this.
Private access to Azure services¶
Azure PaaS services are reached through private endpoints in the private endpoint subnet, with public network access
disabled on the service. Today the solution creates one private endpoint: for the Key Vault (sub-resource vault, zone
privatelink.vaultcore.azure.net). The design also lists container registry, storage and database zones as the pattern for
later services.
In the hub-integrated mode the hub must provide privatelink.vaultcore.azure.net and privatelink.azurecr.io. The
container registry zone is required by the design, but the solution does not create a registry private endpoint. Either the
hub provides one, or it is added to this repository later.
DNS¶
| Aspect | Design |
|---|---|
| Private DNS zones | One zone per service type, linked to the spoke VNet. Hub-owned in the hub-integrated mode. |
| Resolution of private endpoint names | Through the linked zone, with DNS zone groups creating the records. |
| Hybrid DNS | The design requires on-premises resolution through an Azure DNS Private Resolver in the hub, with inbound and outbound endpoints and forwarding rules. The resolver module exists in the repository but is not wired into the environment. |
| Internal application names | Private DNS zone or internal DNS record for the internal gateway. |
See DNS design for how the MCP hub handles DNS.
Ingress¶
The design uses the Kubernetes Gateway API, so the platform is not tied to one ingress controller.
| Owner | Resources |
|---|---|
| Platform team | GatewayClass, shared Gateway, TLS certificates, load balancer and ingress subnet |
| Workload team | HTTPRoute (or GRPCRoute) pointing at its service. No load balancer, certificate or controller configuration |
Both an internal and an external Gateway can exist, bound to the same GatewayClass. TLS terminates at the gateway by
default; passthrough and re-encrypt are options. Replacing the controller means a new GatewayClass and a changed Gateway
reference, and the HTTPRoute objects stay as they are. The controller and the gateway objects are deployed by the platform
GitOps repository. The load testing design names Traefik as the controller of the reference non-production cluster.
Hub-integrated prerequisites¶
The hub and landing zone must provide, before the first run:
- A spoke VNet with the four named subnets, correctly sized.
- Peering between hub and spoke.
- A firewall or routing intent with rules for the AKS required egress.
- Private DNS zones for the Key Vault and the container registry, linked to the spoke VNet, with their resource IDs
available for
hub_dns_config. - Permissions for the pipeline identity of this solution on the spoke subscription: read the VNet and subnets, and create the Network Contributor role assignment on the spoke VNet.
- Inbound: the hub Application Gateway or firewall pointing at the spoke's internal load balancer. The solution exposes no public IP in this mode.
See Prerequisites for the full checklist.
Environment isolation¶
| Level | Mechanism |
|---|---|
| Production versus non-production | Separate clusters in separate spokes, no spoke-to-spoke peering, separate identities, groups and state |
| Between teams and environments in the non-production cluster | Namespaces <team>-dev, <team>-test, <team>-accp, namespace-scoped RBAC, default-deny network policy, quotas and limit ranges (Security policy) |
The design accepts namespace isolation for non-production only. Namespace isolation does not protect against a container escape, which is one of the reasons production runs on its own cluster.