Operations¶
This page is for the platform team. It covers what to watch, what runs on a schedule, how alerting is wired and what to do when something fails. It describes the reference design; your monitoring, on-call set-up and names may differ. Who performs these tasks depends on the operating model (Operating models, Responsibilities).
Verification after a deployment¶
| Check | Expected |
|---|---|
| Liveness | /.backstage/health/v1/liveness returns 200 |
| Readiness | /.backstage/health/v1/readiness returns 200. It returns 503 during start-up, during shutdown and while the Microsoft sign-in credential fails validation. |
| Build ID | The response of the health endpoint carries the header X-Build-Id, which is the pipeline run ID. The deployment smoke test compares it. |
| Sign-in | A test user reaches a populated catalog |
| Ingestion | A known workload, repository, user and group are in the catalog |
| MCP | The tool list of the MCP server is not empty |
The same checks, with the entity or value used, make up the validation list in Onboarding a customer.
Scheduled work¶
The backend runs these tasks. The values are the defaults in the shared configuration and can be overridden per environment.
| Task | Default | Notes |
|---|---|---|
| Azure Resource Graph, Application Gateway listeners | Every hour, timeout 50 minutes (production configuration) | Can be disabled with enabled: false |
| Azure Resource Graph, subscriptions | Set per provider ID in configuration | Needs a schedule when enabled. An enabled provider without a valid schedule fails at start-up. |
| Azure DevOps repositories | Set in the customer configuration (example: every 30 minutes, timeout 3 minutes) | Reads catalog-info.yaml files |
| Users and groups (Microsoft Graph) | Set in the customer configuration | |
| Network inventory refresh | Every 10 minutes, timeout 2 minutes | Logs are queried on demand |
| Security insights | Every hour, timeout 55 minutes | |
| Compliance insights | Every hour, timeout 55 minutes | |
| Cost export configuration | Batches in an off-hours window | See Cost insights setup |
| Cost data load | Data Factory trigger daily at 04:00 W. Europe time |
If a task takes longer than its timeout, it is cancelled and tries again at the next interval. The Azure Resource Graph requests share a rate limiter budget of 10 requests per 5 seconds, so large estates take longer.
Logging and telemetry¶
| Item | Detail |
|---|---|
| Log level | warn in production. The catalog logs at error. The cost and network plugins log at info. |
| Application Insights | The backend sends traces through Azure Monitor OpenTelemetry. Noisy spans are filtered. |
| Agent telemetry | Agent runs are traced with OpenTelemetry. The traces record inputs and outputs, so they can contain the prompts of users. Apply the same access rules as for other logs that can contain user content. |
| Conversations | Agent threads and messages are stored per user in the portal database |
| Audit events | The access check emits audit events (Authentication and access) |
Alerting¶
The reference environments use PagerDuty. This is a choice of the provider; any Azure Monitor action group works in the same way.
| Part | Detail |
|---|---|
| Shared set-up (once) | A team, a schedule with a weekly rotation limited to the configured business hours and days, and an escalation policy with a configurable delay and number of loops. Managed by Terraform in a platform root that deploys on its own. |
| Per environment | A service linked to the shared escalation policy and to the customer's business service, plus an Azure Monitor action group that calls PagerDuty |
| People | Users are listed with the role manager, responder or observer. Users must already have an account. Use manager for global administrators because PagerDuty does not allow a lower role for them. |
| Coverage | Outside the configured hours the schedule is empty. Remove the restrictions for 24/7 coverage. The hours follow the contract and are not defined here. |
| Alert rules | The modules create no alerts unless you pass action group IDs. The Data Factory module has a pipeline failure alert. |
The shared PagerDuty root must be deployed before an environment that refers to its escalation policy. Its pipeline needs the PagerDuty OAuth client ID, secret and account subdomain as repository secrets, and an OAuth app with read and write scopes for services, schedules and escalation policies and read scopes for teams, users and abilities.
Backup and recovery¶
| Item | Detail |
|---|---|
| PostgreSQL | Automatic backup with 7-day retention, no geo-redundancy |
| Key Vault | Soft delete 7 days, purge protection on |
| Catalog | Rebuilt from Azure and the repositories. Losing the catalog database loses the agent conversations and plugin data, not the source of the catalog. |
| Rollback of the portal | Deploy an earlier commit through the dispatch workflow. |
Private endpoint approvals¶
When the provider deploys from its own network, the private endpoint connections of the customer's resources stay pending until the customer approves them. The customer can revoke an approval at any time, which stops deployments. If a deployment fails to reach Key Vault, the registry or the Terraform state, check first whether a connection is pending or revoked and whether the DNS entry exists. See Onboarding a customer.
Troubleshooting¶
| Symptom | Likely cause | Action |
|---|---|---|
| Sign-in fails with "user does not exist" | The sign-in e-mail does not match a User entity | Wait for the next user sync. Check that the Graph application permissions were granted (Prerequisites). |
| Catalog is empty or workloads are missing | Missing tags, missing Reader role, or the provider schedule has not run | Check the tags (Catalog and entity model) and the role assignments |
| Groups missing | Group name normalisation drops groups that do not follow the naming convention | See Catalog and entity model |
| No cost data | Provider Microsoft.CostManagement not registered, export not yet created, pipeline not yet run, or the private connection to SQL missing |
See Cost insights setup. Allow one to two days after set-up. |
| Network or security tables empty | Missing Monitoring Reader or Reader role on the firewall or gateway scope | Prerequisites |
| Agent pull request fails with 403 or 404 | The identity lacks Contribute on the repository, is not in the Azure DevOps organisation, or the GitHub App lacks permissions | AI setup |
| GitHub integration fails at deployment | The private key was stored with real line breaks | Store the key with \n instead of line breaks (Configuration) |
| Gateway reports the portal unhealthy | The probe points to readiness and a startup or credential problem exists | WAF and custom domain |
| First infrastructure run fails on Key Vault or registry | Expected when private endpoints do not exist yet | Onboarding a customer |
| Deployment of Foundry runs a very long time | Foundry creation can take more than two hours | The pipeline timeout is 210 minutes |
| Foundry deployment fails with a capability host error | A capability host of kind "Agents" exists in an invalid state | See below |
Capability host error¶
This applies when Foundry is deployed with bring-your-own resources. The deployment fails with an error about capabilityHostKind. List the capability hosts of the account, delete the one that is reported and create it again with the REST API.
az rest --method get \
--url "https://management.azure.com/subscriptions/<subscription-id>/resourceGroups/<resource-group>/providers/Microsoft.CognitiveServices/accounts/<account-name>/capabilityHosts?api-version=2025-10-01-preview"
az rest --method delete \
--url ".../capabilityHosts/<capability-host-name>?api-version=2025-10-01-preview"
Create it again with a body that sets capabilityHostKind to Agents and customerSubnet to the resource ID of the delegated agents subnet, then read it back until the provisioning state is succeeded.
Updates¶
New versions of the portal reach every environment through the rings (Deployment and environments). Features are not selected per customer; a customer-specific change is a configuration change. Review the release at the hand-over meeting and after major updates.