Prepare Pipelines and Agents (Azure DevOps)¶

The platform deploys itself from self-hosted agents inside the customer's tenant. Before the hub can be deployed you need three things: a Terraform state storage account, the job runners (agent pool and the network around it), and the mcp container image that the pipelines run in. This page describes them for Azure DevOps. For GitHub see the GitHub variant.

flowchart LR
    s["State storage account<br/>(by hand)"] --> i["infrastructure pipeline<br/>job runners, ACR, pool"] --> c["container pipeline<br/>mcp image"] --> h["hub and spokes"]

The technical background is in Provisioning and the deploy flow.

Prerequisites¶

Item	Detail
Identity and service connections	Deployment service principals and the Azure DevOps service connections from Prepare identity
IP plan for the runners	Ranges for the runner virtual network, the agent subnet, the private endpoint subnet, the Container Apps environment subnet and the DNS resolver subnet (`Azure.JobRunner.IpAddressPrefixes`)
Repositories	The code and configuration repositories from Prepare repositories, with the pipeline YAML available
Rights of the service connection	Contributor or Owner on the subscription that holds the runners. The container pipeline also needs the AcrPush role on the container registry.
Rights in Azure DevOps	The Azure DevOps service DevOpsInfrastructure has Manage agent pools and agents rights in the organization. The deploying identity is a user of the organization and may manage the agent pool.
Resource provider	`Microsoft.DevCenter` is registered in the subscription. The pipeline registers the providers it needs, but check this one if the run fails (Troubleshooting).
Quota	Quota for the runner VM size in the chosen region for the Managed DevOps Pool (resource provider `Microsoft.DevOpsInfrastructure`)
Resource group name	Starts with `rg-` (the pipeline checks this)
State storage	See the next section

1. Create the Terraform state storage account¶

The pipelines do not create the state storage account, the container for the spoke state, or the deployment identities. Create them by hand, in the management subscription of each platform environment.

az account set --subscription <management-subscription-id>

$region      = "<azure-region>"                 # for example westeurope
$shortRegion = "<short-region>"                 # for example weu
$customer    = "<short-name>"                   # at most 6 characters

az group create --location $region --name "rg-$customer-tfstate-$shortRegion-001"
az provider register --namespace Microsoft.Storage
az storage account create --name "stlz${customer}state001" `
    --resource-group "rg-$customer-tfstate-$shortRegion-001" `
    --location $region --sku Standard_LRS

# The spoke state container. The hub container is created by the infrastructure pipeline.
az storage container create --name "<spoke-state-container>" `
    --account-name "stlz${customer}state001" --auth-mode login

The names are examples taken from earlier runbooks. A storage account name is lowercase, at most 24 characters and globally unique. Put the values in Azure.TerraformState.Hub and Azure.TerraformState.Spoke in mcp.json, and give the deployment identities Storage Blob Data Contributor on the account, because the backend uses Entra ID authentication (Naming and state).

The spoke state container

No pipeline creates the spoke state container. If you skip this step, the first spoke run fails on the backend (Known issues).

2. Describe the job runners in `mcp.json`¶

Key	Meaning
`Azure.JobRunner.SubscriptionId`, `.ResourceGroupName`, `.Location`	Where the runners are deployed
`Azure.JobRunner.MaximumAgents`, `.Sku`	Maximum number of concurrent agents, and VM size
`Azure.JobRunner.IpAddressPrefixes.*`	Address ranges of the runner network
`AzureDevops.OrganizationName`, `.AgentPool`	The organization and the name of the Managed DevOps Pool. The container jobs use `<AgentPool>-CTR`.
`AzureDevops.ContainerImageName`	The images that the `container` pipeline builds (normally `["mcp"]`)

Details: Config.

3. Run the `infrastructure` pipeline¶

The pipeline deploys, from Bicep in the code repository: the runner virtual network and NAT gateway, an Azure Container Registry, the Managed DevOps Pool, a Container Apps environment and a DNS resolver. It also creates the hub state container. After the apply it runs smoke tests on the pool.

Name and variables differ between versions

The reference configuration repository calls the entry pipeline infrastructure and reads the values from mcp.json and pipelines/variables.yaml. The how-to in the code repository calls the same pipeline initialize (.azure-devops/pipelines/initialize.yaml) and drives it with the variables below. Use the form that your pinned version has.

If the pipeline does not exist yet: Pipelines, New pipeline, Azure Repos Git, the repository, Existing Azure Pipelines YAML file, the branch and the path of the YAML file, Save. Rename it to make the purpose clear.

If your version uses pipeline variables, define them in a variable group in the library and link it to the pipeline. A stage runs only when its selector variable contains the action name.

Variable (per platform environment, prefix `canary_` or `production_`)	Meaning
`canary`, `production` (selectors)	A stage runs when the value contains `initialize`. Examples from the how-to: `initialize`, or `initialize,container`.
`<env>_service_connection`	Name of the Azure service connection
`<env>_subscription_id`	Subscription for the runners
`<env>_location`	Azure region
`<env>_resource_group_name`	Resource group, starting with `rg-`
`<env>_agent_pool_mdp`	Name of the agent pool
`<env>_maximum_agents`	Maximum concurrent agents
`<env>_sku`	VM size of the agents
`<env>_vnet`, `<env>_vnet_snet_agent`, `<env>_vnet_snet_pep`, `<env>_vnet_snet_cae`	CIDR ranges of the virtual network and its agent, private endpoint and Container Apps subnets

Run it for canary first (stage selector for canary only), check the result, then run prod. A clean run takes roughly 10 to 20 minutes per stage.

Verify¶

In the Azure portal, the runner resource group holds the virtual network, the Container Registry, the managed identities, the Managed DevOps Pool and the Container Apps environment.
In Azure DevOps, Project settings, Agent pools: the pool exists and agents come online (give it five to ten minutes after the first run).
The smoke tests passed. They check that the pool exists and is provisioned, has a maximum concurrency and a valid SKU, is registered in Azure DevOps as a managed pool, has the expected images configured, can be queried for agents, and that the subnet, the DevCenter project and the organization profile are configured.

4. Run the `container` pipeline¶

This builds the mcp image from the pinned code version and pushes it to the Container Registry of that platform environment. It needs the infrastructure run to have finished, because it uses the registry and the container pool.

Variable (per platform environment)	Meaning
`agent_pool_mdp`	Pool used by the orchestration stage
`<env>_agent_pool_ctr`	Pool for the container build jobs (`<AgentPool>-CTR`)
`<env>_container_image_refs`	Images and tags to build, as `<image>:<tag>[,<image>:<tag>]` (for example `mcp:latest`)
Service connection, subscription ID, resource group	As for the infrastructure pipeline

Include a version tag as well as latest so every image can be traced to a release. How the tags are set is described in Releasing.

Verify in the portal (Container Registry, Repositories) that the repository and the expected tags exist, or with the Azure CLI: az acr repository list --name <acr-name> and az acr repository show-tags --name <acr-name> --repository mcp.

Rebuild the image after every change of the pinned code version (What to do when a new version is released).

5. Network access of the agents¶

The agents must reach Azure DevOps, the package sources the pipelines install from, and Azure. The hub firewall sits in the way once the hub is deployed. Check these points before you deploy the hub:

Set deployment_agents_private_network_range in the hub configuration so the platform creates the firewall rule collection group for the agents (Firewall rules).
Agents reach the internet through the hub firewall when the hub connection of their virtual network has internet_security_enabled = true. A destroy of the connectivity layer then cuts the agents off from Azure DevOps. Setting internet_security_enabled = false on the connection of the runner network avoids that dependency.
Do not use a catch-all (.) rule in the DNS resolver forwarding rules. Azure-internal host names used by the pool only resolve through Azure DNS. Use specific per-domain rules.
The destinations that older agent rules had to allow: github.com and githubusercontent.com hosts, dev.azure.com and the Azure DevOps agent package and identity hosts, the PowerShell Gallery, the HashiCorp release, registry and checkpoint hosts, and packages.microsoft.com. Treat the list as a starting point and read the failures in the logs. Access from workload teams' own agents is a separate topic: Build agent access to spokes.

Troubleshooting¶

Symptom	Likely cause	Action
"Entra-based Azure DevOps preflight failed"	The service principal or user is not in the Azure DevOps organization, or has no rights on the pool	Add it under Organization settings, Users; give it Administrator or Service Account on the agent pool; rerun
"Resource provider not registered"	`Microsoft.DevCenter` (or another provider) is not registered	`az provider register --namespace Microsoft.DevCenter --wait`, then rerun
A stage is skipped	The selector variable does not contain the action name	Fix the variable and rerun
"Resource group name must start with `rg-`"	The resource group variable breaks the naming rule	Rename it and rerun
"MDP quota preflight failed"	Not enough quota for the VM family	Request a quota increase for `Microsoft.DevOpsInfrastructure`
No agents in the pool	Registration delay, or the agent subnet has no outbound path	Wait; check the firewall, the NAT gateway and DNS (see above); check that the managed identity may register agents
Smoke test: `statuses` null, agents never connected	Firewall blocks the agent subnet, or the NAT gateway is overridden by a hub connection that routes internet traffic through the firewall	Allow the agent subnet outbound on ports 80 and 443, or adjust the hub connection
"Container registry not found" in the container pipeline	The `infrastructure` run did not finish, the stage selector is wrong, or the identity has no AcrPush role	Check the registry in the resource group and the role assignment

Older approach: virtual machine scale set agents¶

Versions before the Managed DevOps Pool deployed the agents with a separate pipeline and repository. You may still meet it in tenants that have not been upgraded.

Three stages in one pipeline: a Shared Image Gallery, an agent image built from Microsoft's self-hosted agent image source and stored in the gallery, and a virtual machine scale set based on that image. Stages 1 and 3 use Terraform.
Two service principals (canary and prod) with Contributor and User Access Administrator on the subscription, and two service connections for them.
A variable group with the service principal ID and secret and the names of the gallery, its resource group and the image.
Terraform variable files for the gallery and for the agents: customer short code, region, VM SKU, instance count, admin user and password, the virtual network and subnet ranges, and the DNS servers of the scale set network (a customer DNS server plus the Azure DNS address 168.63.129.16).
The Terraform extension for Azure DevOps installed in the organization.
The agent pools were created by hand: Project settings, Agent pools, Add pool, type Azure virtual machine scale set, the service connection and the scale set. Typical settings: tear down virtual machines after every use, a maximum of 10 machines, 2 on standby, 30 minutes delay before deleting idle agents, access for all pipelines.
Pipelines selected the pool by name in the pool: section.

When you upgrade such a tenant, run infrastructure and container to create the new runners, point the entry pipelines to the new pool, and decommission the scale set agents afterwards (Migrating from an older platform version).