Container Agent Pool: Azure Pipelines Agents on Container Apps Jobs

This page is for the platform team. It describes the cae-agents addon of the shared services: a self-hosted Azure Pipelines agent pool that runs on Azure Container Apps jobs. Agents start per queued pipeline job, run that one job in a fresh container, and are removed afterwards. When nothing is queued, nothing runs and nothing is paid for.

The workload team's view (how to use the pool and its limits) is on Build agents in the network. The network design for agents is on Build agent access to spokes. Who operates the pool depends on your operating model.

Reference design, your deployment may differ

The facts below come from the Terraform code, the container image files and the README of one reference implementation. Names, CIDR ranges and the set of tools in the image are examples.

Why container-based agents

	Container-based (this addon)	VM-based pool
Cost	Pay for the seconds a job runs. Zero when idle.	VMs run whether or not pipelines run
Scaling	From 0 to a maximum, per queued job	Fixed size or slower VM scale-out
Isolation	A fresh container per job, no state left behind	Shared state unless the VM is re-imaged
Maintenance	No OS patching of VMs. A new image is a new tag.	Patching, reboots or re-imaging
Start time	About 50 seconds when warm, 3 to 4 minutes cold	Immediate for standby agents, slow for new VMs

The container image does not contain a Docker daemon, and the container does not run as root. Workloads that need a Docker daemon or root need another pool (see Shared services).

How it works

flowchart TD
    q["Pipeline job queued in the agent pool"] --> k["KEDA scaler<br/>polls the pool queue every 10 s"]
    k --> j["Container Apps job execution<br/>(one per queued job)"]
    j --> pull["Pull image from the registry<br/>(managed identity, AcrPull)"]
    pull --> s["start.sh"]
    s --> login["az login --identity<br/>token for Azure DevOps"]
    login --> reg["Download agent, register in the pool"]
    reg --> run["run.sh --once: run one job"]
    run --> unreg["Remove the agent from the pool"]
    unreg --> done["Container exits"]

A placeholder agent is registered once after deployment. Without it, the pool does not have the Agent.Version capability that Microsoft-hosted agents have by default, and queued jobs are never matched to an agent.

What the code deploys

Resource	Name pattern	Purpose
Resource group	rg-agent-pool-<workload>-<postfix>	Holds the agent resources
User-assigned managed identity	id-container-agent-<workload>	Pulls the image and talks to Azure DevOps
Role assignment	AcrPull on the registry	Lets the identity pull the agent image
Container Apps environment	cae-agent-<workload>	In the spoke subnet, with an internal load balancer, logs to the spoke Log Analytics workspace. The identity is also set on the environment, because the control plane validates registry access when a job is created.
Container Apps job (agent)	caj-agent-<workload>	Event-triggered. One execution per queued pipeline job.
Container Apps job (placeholder)	caj-placeholder-agent-<workload>	Manually triggered, registers the placeholder agent
Azure DevOps agent pool	container_agent_pool_name	Organisation-level pool. Auto-provisioning and auto-update are off.
Service principal entitlement	the managed identity	Licence level express in Azure DevOps
Security role assignment	the managed identity	Administrator on the agent pool, so the identity can register and remove agents

Settings of the agent job in the code:

Setting	Value
Trigger	KEDA scaler azure-pipelines, polling interval 10 s, target queue length 1 per execution, authenticated with the managed identity
Scale	container_agent_min_replicas (default 0) to container_agent_max_replicas (default 25) concurrent executions
Size	1 vCPU, 2 GiB memory per container
Time limit	replica_timeout_in_seconds = 1800: an execution is stopped after 30 minutes. Retry limit 1.
Workload profile	container_agent_workload_profile, default Consumption
Environment variables	AZP_URL, AZP_POOL and MANAGED_IDENTITY_CLIENT_ID. The placeholder job also sets AZP_PLACEHOLDER, AZP_RANDOM_AGENT_SUFFIX and AZP_AGENT_NAME.

The 30-minute limit applies to every pipeline job on this pool. Tell workload teams, or use another pool for long jobs.

Variables

Variable	Meaning
spoke_config	Resource group, virtual network, subnet for the environment and Log Analytics workspace of the spoke, by name
container_agent_pool_name	Name of the Azure DevOps pool
container_agent_azp_url	URL of the Azure DevOps organisation
container_agent_location and location	Azure region
container_agent_image_tag	Tag of the azp-agent image in the registry
azure_container_registry	Name, resource group and subscription of an existing registry that holds the image
workload_name	Part of the resource names (default ccap)
resource_postfix	Part of the resource group name
container_agent_min_replicas, container_agent_max_replicas	Scale limits (0 and 25 by default)
container_agent_workload_profile	Workload profile of the environment and jobs

Prerequisites

Azure

• A spoke subnet for the Container Apps environment, delegated to Microsoft.App/environments. Without the delegation the environment deployment fails. Size the subnet for the maximum number of concurrent agents.
• A registry with the agent image, with the admin user disabled. The reference needs the Standard SKU at minimum. The pool authenticates only with the managed identity, and the module assigns AcrPull to it.
• The spoke Log Analytics workspace.
• Firewall access, see Network.

Azure DevOps

• A service connection that is allowed to manage agent pools.
• A personal access token for the Azure DevOps Terraform provider, kept in the variable group of the pipeline. It is used only while Terraform runs, to create the pool and give the identity its licence and role, not by the agents at runtime. It needs the scopes Agent Pools (Read & Manage) and Member Entitlement Management (Read & Write). A personal access token expires: add it to your secret monitoring (Secret watcher).
• After the deployment, link the organisation-level pool to each project that uses it: Project settings, Pipelines, Agent pools, Add pool, Existing pool.

Network

In the reference design the spoke is connected to a Virtual WAN hub, and the hub advertises a default route (0.0.0.0/0) to the spoke. All internet traffic of the environment subnet goes through the hub firewall.

Requirement	Detail
Required firewall rule	The CIDR of the environment subnet must be in an IP group used by a rule that allows outbound TCP 443. In the reference this is the IP group of the self-hosted agents, see Build agent access to spokes.
Destinations needed	The Microsoft Entra sign-in endpoint (login.microsoftonline.com), the registry (*.azurecr.io), the Azure DevOps service and its agent download host, and Azure blob storage (*.blob.core.windows.net)
Optional	Service endpoints Microsoft.ContainerRegistry and Microsoft.Storage on the subnet. Image pulls and blob traffic then bypass the firewall, which lowers firewall processing cost and pull latency. Not needed when the firewall rule allows the traffic.
HTTP (port 80)	Blocked by the firewall in the reference design. The image build therefore cannot use apt-get, which uses HTTP by default.

Without the rule, terraform apply fails with unable to pull image using managed identity, because the environment does a live test pull when a job is created. Executions fail with ImagePullFailure.

The rule must be in the deployed firewall policy

Adding the CIDR to the IP group in the configuration repository is not enough. Check the deployed IP group.

The Container Apps environment uses an internal load balancer (internal_load_balancer_enabled = true). Agents only connect outbound to Azure DevOps, so this is not a limitation for the pipeline.

The agent image

The image is built from the files in infra/cae-agents/azp-agent-in-docker/:

Item	Detail
Base	A Microsoft devcontainer base image on Ubuntu 22.04, pulled from the Microsoft Container Registry. It already has curl, git, jq, unzip, sudo, wget, rsync and openssh-client.
Added	tini, Azure CLI (installed with pip) and PowerShell (a pinned version from the release archive), all over HTTPS
User	A non-root user agent
Entry point	start.sh

start.sh signs in with az login --identity using the client ID in MANAGED_IDENTITY_CLIENT_ID, requests a token for Azure DevOps, downloads the latest agent package, registers the agent (--replace), runs it with --once, and removes the agent when the container stops. The identity endpoint of Container Apps is injected as environment variables, and az login --identity uses it, not the VM metadata endpoint. In placeholder mode it registers the agent and exits without running a job.

The Dockerfile in the repository has windows in its name, but it builds a Linux image. The repository also has an older file with the suffix _old.

To update the image: build and push a new tag to the registry, set container_agent_image_tag and apply. Keep the image small, as the cold start includes pulling it (about 490 MB in the reference). The tools in the image are only those above: workload teams install other tools in their pipeline (Build agents).

Deploy

1. Make sure the prerequisites are in place and the image tag exists in the registry.
2. Run the shared services pipeline with deployCAEAgents (Shared services). The plan uses the variables file of the addon. The apply creates the resources.
3. The pipeline then starts the placeholder job (az containerapp job start). If you deploy another way, start it once by hand.
4. Link the pool to the projects (Prerequisites).
5. Run a test pipeline on the pool and check the agent information, the tools and the managed identity token.

Startup time and cost

Scenario	Typical time	Reason
Cold start (first job, node recycled)	3 to 4 minutes	Full image pull and agent download (about 100 MB)
Warm start (node reused)	About 50 seconds	Image layers cached on the node, only registration is new

Consecutive jobs benefit from a warm node. If cold starts are not acceptable, set container_agent_min_replicas to 1. One container then always runs, at a constant cost.

Security

• Every job runs in a fresh container, so credentials and files do not carry over between jobs.
• Agents authenticate to the registry and Azure DevOps with the managed identity, not with keys. The module assigns the identity only AcrPull in Azure. What a job may do in Azure comes from the service connection of the pipeline, as for other pools (Build agent access to spokes).
• The identity is Administrator on the pool. Limit who else has rights on the pool (Azure DevOps).
• For production deployments, use a separate pool, as recommended in the design guidance.

Troubleshooting

Symptom	Likely cause and action
terraform apply fails with unable to pull image using managed identity, or executions fail with ImagePullFailure	The firewall blocks the subnet. Check that the CIDR is in the deployed IP group, for example with az network ip-group show --name <ip group> --resource-group <hub resource group> --subscription <hub-subscription-id> --query ipAddresses. Check the logs below.
No agents appear in the pool after the placeholder job	Check the status of the last execution of the placeholder job. If it failed, check the logs. If it succeeded, check that container_agent_azp_url has no trailing slash and that the pool is linked to the project.
The pool does not exist or has not been authorized for use	The pool is not linked to the project of the pipeline
Jobs stay queued and are never assigned	The placeholder agent is missing, so the pool has no Agent.Version capability. Start the placeholder job.
An execution is stopped after 30 minutes	The time limit of the job. Shorten the job or use another pool.
A job cannot start, the job is suspended	Azure suspends a Container Apps job after repeated failures. Fix the cause, then resume it, see below.

Query the system logs of the environment in the spoke Log Analytics workspace:

ContainerAppSystemLogs_CL
| where TimeGenerated > ago(30m)
| project TimeGenerated, Reason_s, Log_s
| order by TimeGenerated asc

Resume a suspended job:

az rest --method POST \
  --url "https://management.azure.com/subscriptions/<subscription-id>/resourceGroups/<resource group>/providers/Microsoft.App/jobs/<job name>/resume?api-version=2023-05-01"

Related

• Build agents in the network
• Build agent access to spokes
• Shared services: what is deployed and how
• Platform: deployment tooling and job runners (the platform's own runners)