Troubleshooting How-Tos

Step-by-step guides for problems the platform team meets in the pipelines, the build agents and local development. For a first look at an incident, see Where to look first. For defects that are known, see Known issues.

Problem	Go to
A plan or apply fails or hangs without a clear error	Turn on Terraform logging
Build agents lost their route to Azure DevOps after a destroy	Restore broken agent connectivity
Pipeline jobs wait for agents, or agents never register	Check the agent pool with the smoke tests
You cannot reach private resources from your development machine	Reach private resources from a dev container

Turn on Terraform logging

Use this when a Terraform plan or apply step fails or hangs without a clear error, when you need to see the API calls a provider makes, or when support asks for Terraform debug logs.

Terraform has two log settings that you can turn on separately:

Setting	Environment variable	What it shows
Core log level	TF_LOG_CORE	Terraform's own log: graph building, plan and apply orchestration
Provider log level	TF_LOG_PROVIDER	The log of the provider plugins (for example azurerm), including the requests and responses they make

The values are TRACE, DEBUG, INFO, WARN and ERROR. A pipeline parameter with the value DEFAULT leaves the variable unset, so logging behaves as normal.

Start with the core log at DEBUG: it is usually enough to see what Terraform is doing. Use the provider log only for a short time and only for a specific problem.

Provider logs can contain secrets

TF_LOG_PROVIDER at DEBUG or TRACE can print unmasked secrets from provider request and response payloads. Turn it back to DEFAULT as soon as you have what you need. Treat the logs of that run as sensitive: restrict access and delete them when they are no longer needed.

Azure DevOps

The older environment pipelines (pipelines/environment_canary.yaml and environment_prod.yaml) expose two run parameters, terraformLogCore and terraformLogProvider (default DEFAULT):

1. Open the pipeline and select Run pipeline.
2. In the parameters, set Terraform core log level or Terraform provider log level.
3. Run it. The plan and apply jobs set TF_LOG_CORE and TF_LOG_PROVIDER for the length of the run.

GitHub Actions

The reusable workflows environment_canary.yml and environment_prod.yml accept the same two inputs. Pass them from the calling workflow:

jobs:
  deploy:
    uses: <organisation>/<code repository>/.github/workflows/environment_canary.yml@<version>
    with:
      environment: canary
      regions: '["weu"]'
      deployList: ${{ inputs.deployList }}
      defaultRunnerGroup: <runner group>
      terraformLogCore: DEBUG
      terraformLogProvider: DEFAULT

Check your entry pipeline

These parameters are documented for the environment pipelines of the code repository. Open the template that your entry pipeline (hub, spoke) calls and look for terraformLogCore and terraformLogProvider. If they are not there, ask the owner of the code repository how to enable Terraform logging for your version.

Verify the logging

In the Terraform Plan or Terraform Apply job log (Azure DevOps) or step output (GitHub), Terraform log lines with the level in brackets, such as [DEBUG], appear next to the normal output.

If no extra output appears, both parameters are still DEFAULT, or the value is not one of the levels above. Run again with an explicit level.

Restore broken agent connectivity after a destroy

The problem

When the connectivity layer is destroyed, the virtual hub connection between the agents virtual network and the hub is removed. If the connection has internet_security_enabled = true, all internet traffic of the agents goes through the hub firewall, and that route disappears with the connection. The agents then lose their path to Azure DevOps and stop reporting job output, so the run that did the destroy can appear to hang.

Prevent it

Set internet_security_enabled = false on every hub connection whose network hosts pipeline agents. The agents then reach Azure DevOps directly over the internet, independent of the hub. In environments/<env>/<region>/connectivity.auto.tfvars:

virtual_hub_connections = {
  <connection name> = {
    internet_security_enabled = false   # agents use direct internet, not the hub firewall
    remote_virtual_network_id = "/subscriptions/<subscription-id>/resourceGroups/<resource-group>/providers/Microsoft.Network/virtualNetworks/<agents vnet>"
    routings                  = null
    virtual_hub_id            = null
  }
}

See also the setting for spokes in the Spoke JSON reference (InternetSecurityEnabled). The agents of the platform itself run in the job runner network, see Provisioning.

Recover without a Terraform apply

If connectivity is already broken, change the connection directly through the Azure REST API. The az network vhub connection update --internet-security option is not supported in older Azure CLI versions, and the resource provider does not accept PATCH on hub connections. Read the connection, change it, and write it back (GET and PUT).

1. Find the virtual hub and its resource group.

   az network vhub list `
     --subscription <connectivity-subscription-id> `
     --query "[].{name:name, rg:resourceGroup}" `
     -o table
   

2. Check that the connection exists.

   az network vhub connection list `
     --vhub-name <vhub-name> `
     --resource-group <resource-group> `
     --subscription <connectivity-subscription-id> `
     --query "[].{name:name, internetSecurity:properties.enableInternetSecurity}" `
     -o table
   

3. Disable internet security with GET and PUT.

   $url = "https://management.azure.com/subscriptions/<connectivity-subscription-id>/resourceGroups/<resource-group>/providers/Microsoft.Network/virtualHubs/<vhub-name>/hubVirtualNetworkConnections/<connection-name>?api-version=2023-09-01"
   
   $conn = az rest --method get --url $url | ConvertFrom-Json
   $conn.properties.enableInternetSecurity = $false
   az rest --method put --url $url --body ($conn | ConvertTo-Json -Depth 20 -Compress)
   

4. Wait for the update to finish.

   do {
       Start-Sleep -Seconds 10
       $state = az rest --method get --url $url | ConvertFrom-Json
       Write-Host "$(Get-Date -Format 'HH:mm:ss') - provisioningState: $($state.properties.provisioningState)"
   } until ($state.properties.provisioningState -ne 'Updating')
   

5. Persist the fix in code. Set internet_security_enabled = false in the connectivity.auto.tfvars of the affected region, so the next apply does not undo your change (Change workflow). A change in Azure without a matching change in code is drift (Incident management).

Check the Managed DevOps Pool with the smoke tests

The platform's own job runners run on a Managed DevOps Pool (MDP) (Provisioning). The smoke tests check that the pool is deployed correctly and can start VMs and register agents with Azure DevOps. Run them after every infrastructure deployment, and when jobs wait for agents.

What they check

Area	Checks
Azure Resource Manager	The pool exists in the resource group. Its provisioning state is Succeeded. The maximum concurrency is above 0. The SKU is valid. Status information is available (a null status points at network problems)
Azure DevOps registration	The pool is registered in the organisation, has the type automation (managed pool) and is marked as hosted
Images	The expected images are configured: Windows Server 2025 and 2022, Ubuntu 24.04 and 22.04 (each as <image>/latest)
Agent connectivity	Agents can be queried from the Azure DevOps API, and none is stuck in the never connected state (a sign of a failed bootstrap)
Networking	The agent subnet is configured and referenced. The DevCenter project exists and is linked. The organisation profile points to the right Azure DevOps organisation. Parallelism is configured

Run them

The smoke tests run automatically after Invoke-XebiaInfrastructure -Action Apply, which the infrastructure pipeline calls. To run them separately, for example when you troubleshoot, use the dev container (Run the tests locally):

# In the dev container, from the root of the code repository
Import-Module ./src/Xebia.Mcp/Xebia.Mcp.psd1 -Force
Initialize-XebiaDotEnv
$config = Resolve-XebiaMcpConfiguration -Path $env:MCP_CONFIG
Test-XebiaMcpSmoke -Config $config

You need to be signed in with az login and have access to the subscription of the pool, plus the Agent Pools (Read & manage) permission in Azure DevOps. The result is shown on the console and written as an NUnit XML file (artifacts/testresults-smoke.xml), which a pipeline can publish.

For verbose output, set $VerbosePreference = 'Continue' before the call.

Common failures

Symptom	Cause	Fix
statuses: null or resources: []	The firewall blocks the CIDR of the agent subnet	Add the agent subnet to a firewall IP group with outbound allow rules for port 443 (and port 80 if your agents need plain HTTP). Check ip_groups/ and firewall_rule_configuration/ of the region (Build agent access)
agentConnectedTime: null and receiveTime: null	The NAT gateway is off, or the firewall blocks the agents' outbound traffic	Enable the NAT gateway on the agent subnet, or allow the agent subnet to the internet on the firewall. If the network is connected to the hub with internet security on, all internet traffic goes through the firewall and the NAT gateway is bypassed
Agents start but cannot resolve Azure host names	A catch-all forwarding rule (.) in the DNS resolver sends every query to a public DNS server. Azure-internal names that the pool uses only resolve through Azure DNS (168.63.129.16)	Remove the catch-all rule. Create only forwarding rules per domain, never a catch-all (DNS design)
MDP quota preflight failed	Not enough quota for the SKU family	Request more quota from Azure support for Microsoft.DevOpsInfrastructure

Go further

• Read the pool resource directly with az rest against Microsoft.DevOpsInfrastructure/pools/<pool> of the job runner resource group (API version 2024-10-19).
• List the agents with the Azure DevOps REST API: get the pool ID from _apis/distributedtask/pools?poolName=<pool>, then read _apis/distributedtask/pools/<id>/agents?includeCapabilities=true, with a token for the Azure DevOps resource.
• End-to-end test. Queue a small test pipeline that runs one job per image on the pool. Each job should get an agent within 5 to 10 minutes, finish, and the agent should be deallocated afterwards.

Reach private resources from a dev container

Spoke work often needs access to private resources (Key Vaults, storage accounts) from a development container. This guide connects a Windows machine with WSL2 and Docker Desktop to the hub over point-to-site VPN, and fixes DNS inside the container so private names resolve.

Before you start

• A Windows host with WSL2 and Docker Desktop, and local admin rights to install the Azure VPN Client.
• Access to the subscription and resource group of the point-to-site VPN gateway, and permission to generate VPN client profiles.
• The Xebia.Mcp module in the dev container.
• A point-to-site VPN gateway in the hub. It is optional, off by default, and set up with a standard change (Enable point-to-site VPN below, and Developer access).

1. Check the point-to-site gateway

In the Azure portal open the point-to-site VPN gateway in the connectivity resource group, then Point-to-site configuration. Verify the client address pool, the authentication method (for example Microsoft Entra ID or certificates) and the tunnel type. If platform automation configured it, check the settings only.

2. Download the VPN client profile

From the portal: in the gateway, select Download VPN client and save the zip. Or from the dev container:

$profile = Invoke-XebiaAzureCli -ScriptBlock {
    az network p2s-vpn-gateway vpn-client generate `
    --subscription '<subscription-id>' `
    --resource-group '<resource-group-name>' `
    --name '<p2s-gateway-name>'
} -WhatIf:$false | ConvertFrom-Json

Invoke-WebRequest -Uri $profile.profileUrl -OutFile ./vpn-client-profile.zip
Expand-Archive -Path ./vpn-client-profile.zip -DestinationPath ./vpn-client-profile -Force

3. Install the Azure VPN Client and import the profile

1. Install Azure VPN Client from the Microsoft Store, or through your enterprise software distribution.
2. Start it and select Import. Choose the zip, or browse to the extracted AzureVPN/VpnSettings.xml.
3. Select the imported profile and Connect. Sign in if asked. The status must be Connected.

Quick checks on Windows:

Resolve-DnsName '<private-resource-name>'
Test-NetConnection '<private-ip-or-fqdn>' -Port 443

Test-NetConnection only confirms that TCP port 443 is open. To confirm that the endpoint presents the expected service certificate, run the TLS check in Verify the connection.

4. Configure WSL networking

Update WSL from Windows PowerShell with wsl --update and check wsl --version. Create or edit %UserProfile%\.wslconfig:

[wsl2]
networkingMode=mirrored
dnsTunneling=true
firewall=true
autoProxy=true

Apply it: close all WSL terminals, run wsl --shutdown in Windows PowerShell, reconnect the Azure VPN Client, then reopen WSL and the dev container.

5. Point the container at the hub DNS

If names in the container still resolve to public addresses, force the hub DNS server temporarily:

Import-Module Xebia.Mcp -Force
Set-XebiaLocalDns -NameServer <hub DNS server IP>

When you finish the task, restore the original DNS:

Set-XebiaLocalDns -Restore

Set-XebiaLocalDns keeps one backup file of resolv.conf (/tmp/xebia-localdns-resolv.conf.bak). Running it again with -NameServer while the backup exists is blocked on purpose: restore first. -Restore restores the file and deletes the backup.

Verify the connection

Run these inside the dev container after the VPN is connected. Replace the vault name and the private IP with those of the resource you test.

# 1. The name resolves to the private endpoint address
[System.Net.Dns]::GetHostAddresses('<vault-name>.vault.azure.net') | Select-Object -ExpandProperty IPAddressToString

# 2. TCP port 443 is open
$tcpClient = [System.Net.Sockets.TcpClient]::new(); $tcpClient.Connect('<private-ip>', 443); "TCP OK"; $tcpClient.Dispose()

# 3. The endpoint presents the expected certificate
$tcpClient = [System.Net.Sockets.TcpClient]::new('<private-ip>', 443)
$callback = [System.Net.Security.RemoteCertificateValidationCallback] { param($sender, $certificate, $chain, $errors) $true }
$sslStream = [System.Net.Security.SslStream]::new($tcpClient.GetStream(), $false, $callback)
$sslStream.AuthenticateAsClient('<vault-name>.vault.azure.net')
$sslStream.RemoteCertificate.Subject
$sslStream.RemoteCertificate.Issuer
$sslStream.Dispose(); $tcpClient.Dispose()

# 4. A real call
az keyvault secret list --vault-name <vault-name> --maxresults 1 -o table

Expected: the name resolves to the private IP, TCP OK is printed, a Microsoft-issued certificate for vault.azure.net is shown, and the last command does not fail with SSL: UNEXPECTED_EOF_WHILE_READING. An empty vault gives empty output, which still proves the network and TLS path. If the TLS check fails with an EOF or returns no certificate, DNS points to a private IP but the container does not reach the private endpoint.

Problems

Symptom	What to do
The VPN is connected but private names resolve to public addresses	Check the Windows VPN is connected. Check that mirrored networking is active after wsl --shutdown. Run Set-XebiaLocalDns -NameServer <hub DNS server IP> in the container
Set-XebiaLocalDns says the backup already exists	Run Set-XebiaLocalDns -Restore, then set the new name server
No route to the private DNS server from the container	Check that the VPN profile routes include the hub and the private DNS ranges. Reconnect the profile. Restart WSL and reopen the container
DNS resolves to the private endpoint and TCP 443 is open, but TLS fails with EOF	See below

TLS fails with EOF while TCP succeeds, from Windows and from the container. This is not only a WSL problem: the TCP connection opens but the TLS handshake with the Azure service does not complete. Check, in this order:

1. The private endpoint IP does not overlap with a local or corporate network. For example, a local network 192.168.123.0/24 conflicts with a spoke private endpoint range inside it.
2. The VPN profile routes the whole spoke virtual network range, not only the DNS server range.
3. DNS resolves the name to the private endpoint IP from both Windows and the container.
4. The private endpoint connection is approved and points to the expected resource.
5. If it still fails, escalate to the network owner with the DNS result, the source IP on Windows, the VPN interface name, the private endpoint IP and the TLS EOF error.

Enable the point-to-site VPN feature

The platform can create a point-to-site VPN gateway in the hub. It is a standard change: Update platform features (Platform features). The steps for this feature:

1. In the configuration repository, create a branch. In <env-folder>/<region>/connectivity.auto.tfvars of the environment:
   • Set create_point_to_site_vpn_gateway = true.
   • Set point_to_site_vpn_gateway_connection_configuration: a name that follows your naming convention (the older runbook uses vpnconf-<company>-connectivity-<env>-<region>-001), and a vpn_client_address_pool with address_prefixes (the CIDR for the VPN clients) and internet_security_enabled (the older runbook sets true).
   • Set entra_tenant_id to the tenant ID of the organisation.
   • Set point_to_site_vpn_enterprise_app_id to the application ID of the Azure VPN application (see the note below).
2. Check the companion variables in variables_point_to_site_vpn_gateway.tf (the DNS server of the gateway must equal virtual_network_dns_servers, as a config test requires).
3. Run hub in Plan mode for the changed environment and region. Only the connectivity layer should show changes. Apply to canary, then prod, with approval for prod.
4. Verify in the portal: Virtual WAN > Hubs > the hub of the region > Point-to-site configuration. Download the configuration file and test the connection with Azure VPN Client.
5. Merge the pull request.

To use the VPN for development, continue with Reach private resources from a dev container. For forced tunnelling and the design of per-workload access, see Developer access.

The Azure VPN application ID

The older runbook uses 41b23e61-6c1e-4545-b367-cd054e0ed4b4 for point_to_site_vpn_enterprise_app_id. Check it against Microsoft's current documentation for Azure VPN with Entra ID authentication before you use it, because the value depends on the cloud.

Related pages

• Known issues
• Incident management
• Run a single layer locally