Skip to content

Build Agents in the Network

Every Azure DevOps job (a build, a deployment, anything else) runs on an agent. Microsoft-hosted agents are managed by Microsoft, but they run outside your network and can't reach private endpoints. A self-hosted agent pool inside the platform network can. This page explains how such a shared pool is usually built (a VM-based and a container-based example), how you use it, and what to do when the OS image of the pool changes.

Platform-specific

Whether your platform offers a shared agent pool, its name, its size and its OS image depend on your platform. The pool design below is an example. The pools that the MCP pipelines use for the platform itself (see Job runners) are for platform deployments. Ask your platform team which pool you may use. The short summary is on Shared services. The platform team runs the pool, or you run it yourself in the self-managed model (Operating models).

Example design of a shared pool

Topic Example
Compute A Virtual Machine Scale Set in the shared services of the hub, attached to Azure DevOps as an agent pool
Image Built in a private Shared Image Gallery and refreshed on a schedule (for example every Sunday) from the same source as the Microsoft-hosted agent image. The pool offers the same tools as the Microsoft-hosted agents, list of installed software.
OS Linux only, in the example
Lifecycle Agents are torn down after every use, so no secrets or certificates stay behind
Scaling A maximum number of VMs in the scale set, a number of agents kept on standby, and a delay before idle agents are removed
Network A dedicated subnet in the hub's shared services network, connected to the spokes (through Virtual WAN)
Firewall HTTPS (and SQL) from the agent subnet to Azure destinations. Everything else needs a firewall rule.
Access to the Azure resources of the pool Only the platform team
Failure of the image pipeline Existing agents keep working. New images are not rolled out until the pipeline is fixed. The usual cause is a change in the upstream image repository.

Security model. The pool is available to all connected workload teams, but a job can only do what the service principal of your pipeline may do. That service principal is created with your spoke and has the rights of your spoke (What you get). The first time a pipeline uses a service connection, your team has to authorise it, so a pipeline of another team can't use your service principal. Because the agent is torn down after use, credentials can't be recovered from it afterwards.

Use the pool

  1. Ask your platform team for access to the pool, and the pool name. If your pipeline lives in another Azure DevOps project than the one that owns the pool, the pool must also be made available to that project.
  2. Set the pool in your YAML pipeline:
trigger:
  batch: true
  branches:
    include:
      - main

pool: <agent pool name>

variables:
  - group: "<variable group>"

stages:
  # your stages
  1. Use a service connection that belongs to your spoke (Azure DevOps).

Pool types and their limits (VM-based and container-based) are described in Shared services.

Example: container-based pool on Container Apps jobs

A container-based pool runs each pipeline job in its own container on Azure Container Apps jobs. A scaler watches the queue of the Azure DevOps pool and starts one container per queued job. The container registers as an agent, runs that one job, removes itself from the pool and stops. With nothing queued, nothing runs.

Topic Example
Lifecycle One fresh container per job. No files or credentials remain afterwards.
Scaling From 0 to a maximum number of containers (25 in the example), the queue is checked every 10 seconds
Start time About 50 seconds when the node is warm, 3 to 4 minutes when it is cold (first job after a quiet period)
Size 1 vCPU and 2 GiB of memory per job
Time limit A job is stopped after 30 minutes in the example
Identity The container uses a managed identity to pull its image and to register in Azure DevOps. What your job may do in Azure comes from your service connection, as on any other pool.
Network In a subnet of a spoke. Outbound traffic goes through the hub firewall.
OS Linux, as a container image

Cold starts are the price of scale-from-zero. If your platform keeps one agent running, the first job starts faster, at a constant cost for the platform.

Tools in the container image

The example image is small. It is based on Ubuntu 22.04 and contains the Azure CLI, PowerShell, git, curl, jq, unzip, wget, rsync and an SSH client. The agent runs as a non-root user and there is no Docker daemon. This differs from the VM-based pool, which has the tools of the Microsoft-hosted image.

  • Install what you need in the pipeline with the tasks from Install missing tools in the pipeline. The agent can only download from destinations that the firewall allows (HTTPS 443 in the example), so a download from another port or a plain HTTP URL fails. Ask for a firewall rule if you need another source.
  • Build container images without a Docker daemon: see the options in Shared services.
  • Keep jobs short. Split a long job into stages, because a job that runs longer than the time limit is stopped.
  • Pin the versions of your tools in the pipeline, because the image of the pool can change.

Use a container-based pool

  1. Ask the platform team for the pool name and for access. The pool exists at the organisation level, so it must be linked to your Azure DevOps project (Project settings, Pipelines, Agent pools, Add pool, Existing pool).
  2. Set the pool in your pipeline, as in Use the pool. A first run can take a few minutes because of the cold start.
  3. Use a service connection that belongs to your spoke.
Symptom Cause
The pool does not exist or has not been authorized for use The pool is not linked to your project, or your pipeline is not allowed to use it
The job waits a long time before it starts A cold start (3 to 4 minutes), or the maximum number of containers is in use
The job stops after 30 minutes The time limit of the job
A tool is missing Install it in the pipeline

How the platform team deploys and operates this pool: Container agent pool.

When the OS image of the pool changes

The OS of the image eventually goes out of support, and the platform team replaces it (for example Ubuntu 20.04 by 24.04). The tools in the new image are other versions, and some tools are no longer installed. Pipelines that rely on them break. The platform team should announce the change, with the date, the list of differences and a way to test.

What you do:

  1. Read the announcement and the list of differences. The software lists of the Microsoft-hosted images are on GitHub.
  2. Run your pipelines against the new image before the switch if the platform team offers a test pool or a test date.
  3. Fix pipelines that use a tool or a version that is missing: install it in the pipeline before you use it.

An example of what changed in one real image upgrade (Ubuntu 20.04 to 24.04):

Area Change
Runtimes Node.js 18 to 20, Python 3.8 to 3.12, Ruby 2.7 to 3.2, Perl, Bash, GNU compilers newer
Removed from the image Terraform, NuGet, Mono and MSBuild, Erlang, Sbt, Heroku, some CLI tools (Netlify, Vercel, OpenShift, ORAS, Alibaba Cloud), PhantomJS, R, SVN, MongoDB, sqlcmd and SqlPackage
Tools with a new version Ansible, Bicep, Docker, Podman, Pulumi, OpenSSL, PostgreSQL, browsers
.NET Fewer SDK versions installed (for example only 8.0 on the 24.04 image at that time)

Treat this table as an illustration. The list of the image you actually get is on the GitHub page of the image (Ubuntu2404-Readme.md for Ubuntu 24.04).

Install missing tools in the pipeline

Tool Task Notes
Node.js UseNode@1 Set version, for example '18.x'
Python UsePythonVersion@0 Downloading Python versions is not supported on self-hosted agents. Only pre-installed versions can be selected.
NuGet NuGetCommand@2 To restore project or solution dependencies
.NET SDK UseDotNet@2 Set version, for example '6.x'
Terraform TerraformInstaller@1 Set terraformVersion to latest or to a specific version 
- task: UseNode@1
  displayName: 'Install Node.js'
  inputs:
    version: '18.x'

- task: UseDotNet@2
  displayName: 'Install .NET SDK'
  inputs:
    version: '6.x'

- task: ms-devlabs.custom-terraform-tasks.custom-terraform-installer-task.TerraformInstaller@1
  displayName: 'Install Terraform'
  inputs:
    terraformVersion: '<version>'

If your team uses shared pipeline templates for Terraform, check whether they have a parameter for the Terraform version. Pin the version in your pipeline once the new image is live, so that a later image does not change it unnoticed.

For platform teams: running the pool

  • Announce OS image changes well before the end of support, with the table of differences (generate it from the two upstream software lists) and a test pool or a test window.
  • Monitor the image pipeline, and the size and queue times of the pool. In the example design there was no active monitoring: check that it is adequate for your teams.
  • Start with simple scaling values and adapt them when more teams use the pool.
  • Review the firewall rule for the agent subnet when the set of spokes changes.
  • Document who may use the pool and how a team gets access.
  • For a container-based pool, change the image by pushing a new tag, and announce the changes in the tools as for a VM image. The agent list of the example image is short, so the differences are small. See Container agent pool.