Known Issues

Limitations found while writing this guide, with workarounds. For how to handle an incident, see Incident management.

Verification status (15 October 2026)

The items under Fixed (pending merge) were fixed on a working branch while the guide was written, but they are not in main or release/v4.12 of xms-cloud-platform-code yet. Checked on main (0426660) and release/v4.12 (534ff80):

• dev/dev.ps1 still has the old ValidateSet (Deploy, Unlock, DeploySpoke, …), while its handlers are named HubDeploy, HubUnlock, SpokeDeploy, …. -Action HubDeploy is rejected by parameter validation, and -Action Deploy finds no handler. The spoke handlers are hard-coded to patrick/canvas_nonprod_dev_str.json. Until the fix is merged, call the functions directly (see Run a single layer locally).
• .azure-devops/pipelines/spoke-destroy.yaml still destroys in the order dev → prd.
• The spoke schema environment is still an enum (no acc<n> pattern).

Treat every row under Fixed (pending merge) as open until you have checked it in the version you pin.

Fixed (pending merge)

Issue	Fix
The spoke apply used the literal string spoke_${stateKeyId}.tfstate as the state key, because single quotes stop PowerShell from interpolating.	Invoke-XebiaMcpSpokeApply.ps1 now uses double quotes. A unit test was added.
The spoke schema rejected numbered acceptance environments (acc1, acc2, …), although the pipeline groups them under acc.	spokeschema.json: environment now accepts dev, tst, test, acc, acc<n>, prd, prod, shrd and shared (regex pattern).
dev/dev.ps1 listed action names (Deploy, Unlock, DeploySpoke, …) that had no handler, so the hub/spoke actions could not run. The spoke file was hard-coded.	The ValidateSet now matches the handlers (HubDeploy, SpokeDeploy, …), and a -SpokeFilePath parameter was added.
Azure DevOps hub stage: the firewall job's condition read job_deploy_1_connectivity_apply, which was not one of its dependencies. The identity and management jobs had the same problem in their single-region fallback.	stage-mcp-hub.yaml: the three level-3 jobs now depend on both connectivity apply jobs and use the same condition, matching the GitHub workflow.
Azure DevOps spoke-destroy.yaml destroyed dev → prd, although its header (and the GitHub workflow) says prd → dev.	The order is now prd → acc → tst → dev. Configuration repositories have their own copy of this entry file, so check yours.
Get-XebiaOrderedDeployments documented 3 levels and rejected -Level 4 (postdeployment).	Help text and ValidateSet updated.
Hub approvals: the analyse step kept only the last region's plan summary, so changes in other regions could pass without approval.	Invoke-XebiaHubAnalyseAction.ps1 now aggregates all plan files. A unit test was added.
Azure DevOps hub analyse job downloaded <layer>_$(region)_<level>_json, but that job is not matrixed, so $(region) was undefined.	job-mcp-hub.yaml now downloads <layer>_*_<level>_json/** and analyses every region, matching GitHub.

Open: pipelines

Issue	Impact	Workaround
Entry pipelines hard-code mode and whatIf. In the reference configuration repository, spoke.yaml / spoke-destroy.yaml use whatIf: true and hub.yaml uses mode: Apply, whatIf: false.	Spokes are only planned; the hub applies without approval.	Edit the entry files or expose the parameters (how).
The spoke subscription layer is applied during Plan, even with whatIf: true.	A plan-only run can still change the subscription layer. Today that layer only creates a subscription alias when azureSubscriptionId is empty.	Always set azureSubscriptionId.
Subscription creation needs CSP billing values (xmscsp_*), which the pipeline passes as empty strings.	The spoke pipeline cannot create subscriptions.	Create subscriptions first (New spoke).
Spoke destroy never cancels the subscription, and does not revert its tags or name.	The subscription keeps running.	Decommission by hand (Delete spoke).
A spoke whose environment is not in the pipeline's environments list fails the config test when the glob matches it (for example shared with the default dev/tst/acc/prd).	A broad spokeGlob fails.	Use a narrow spokeGlob, or add shared to environments.
ghrunners and policy_exemptions layers are not part of the current hub orchestration.	Their tfvars have no effect.	Policy exemptions go through postdeployment (Policy assignments).
ADR-0009 says the firewall plan stays concurrent, but Invoke-XebiaHubPlanAction.ps1 also plans the firewall with -parallelism=1.	Slower firewall plans.	None needed.

Open: spokes

Issue	Impact	Workaround
The spoke's azurerm_management_group_subscription_association has a count that depends on an external check: "is the subscription already in the group?" (spoke-canvas/modules/subscription_managementgroup/main.tf). After the first apply the check returns true, so the next plan wants to destroy the association. The run after that recreates it.	Routine spoke re-runs can move the subscription out of its management group and back.	Read every spoke plan. Don't apply a destroy of this resource. A code fix is needed: decide the association once, or import it, instead of using a data-driven count.
Spoke approval (Approve* modes) is decided per spoke-environment stage over every plan summary downloaded in the run, including subscription-layer plans and earlier environments.	Approval prompts are broader than one spoke.	Run one spoke per run (spokeGlob = one file).
Hub config tests (firewall, App Gateway, WAF, DNS baseline) also run in the spoke Prepare step.	A broken hub JSON blocks spoke deployments.	Keep hub JSON valid on the branch you deploy from.
No pipeline creates the spoke state blob container.	The first spoke run fails on the backend.	Create the container by hand (bootstrap).
A Key Vault destroyed with purge protection is recovered on re-create (recover_soft_deleted_key_vaults = true).	Old secrets come back.	Be aware of this when you re-create a deleted spoke.

Open: reference configuration repository entry files

These were found in the reference configuration repository and may also apply to yours:

Issue	Impact	Workaround
pipelines/hub-destroy.yaml and pipelines/spoke-destroy.yaml use template: ./templates/stage-mcp-*.yaml@mcp-code and don't pass agentPoolMdp. The other entries use .azure-devops/pipelines/templates/…@mcp-code and pass it.	The destroy pipelines probably fail to compile.	Use the same template path and parameters as hub.yaml / spoke.yaml.
spoke.yaml targets environments/prod/mcp.json; spoke-destroy.yaml (ADO and GitHub) targets environments/canary/mcp.json.	A spoke deployed with spoke can't be destroyed with spoke-destroy.	Point both at the same mcp.json, or add an environment parameter.
spoke-destroy.yaml runs dev → prd.	Destroy order is not the reverse of deploy order.	Use prd → acc → tst → dev, as the code repository's entry file now does.
tfvars paths are written from the repository root (for example ip_group_json_file_relative_path = "environments/prod/gwc/ip_groups/"), but Terraform joins them to config_path, which is already <repo>/environments/prod (how paths resolve).	Verified in code: config_path is the folder that holds mcp.json (Resolve-XebiaMcpConfiguration passes Split-Path -Parent of the mcp.json path). environments/prod/gwc/ip_groups/ therefore resolves to <repo>/environments/prod/environments/prod/gwc/ip_groups/, which does not exist, so the firewall, policy and App Gateway folders find no files.	Make the paths relative to the environment folder, for example gwc/ip_groups/ and policies/definitions, as in config/Demo/canary of the code repository. Plan first and confirm the expected resources show no changes.
postdeployment.auto.tfvars sets security_contacts = {…}. The current code declares security_contact_email (string, default CustomerSupport@Xebia.com).	The setting is ignored with only a warning, and the default contact is used.	Set security_contact_email.

Open: configuration and validation

Issue	Impact	Workaround
A wrong folder path in tfvars makes fileset() return nothing, without an error.	The plan deletes all resources from that folder.	Read the plan summary; see relative paths.
No IP overlap or containment checks for spoke address ranges.	Overlapping spokes deploy without a warning.	Keep an IP plan and review it in the pull request.
management_group_structure.*.subscription_ids associations are indexed by list position, with ignore_changes.	Removing or reordering entries moves the wrong subscription.	Append only (details).
Spoke Entra ID groups ignore later changes to owners / members.	Editing ad_*Members after creation has no effect.	See Change spoke.
The spoke schema rejects keys that the Terraform reads: extensions.github, extensions.integrations, extensions.apim.workspace, extensions.eventhub.eventhubs, the Service Bus group fields, and azdo: null.	These features can't pass the config test.	Leave azdo out instead of setting it to null. The extension features need a schema update in the code repository. What each extension does: Spoke extensions.
App Gateway config tests only check files named application_gateway_hub.json.	Other gateway files are not validated.	Use that file name.
Two diverged copies of the firewall, App Gateway and WAF schemas exist: scripts/pr-validation/schemas/ and src/Xebia.Mcp/Files/json/. The pipelines use the src copy. The scripts/pr-validation/*.ps1 validators expect an environments/{CANARY,PROD} layout and are not called by any pipeline.	Validator results differ.	Trust the pipeline config tests (src copy).
The firewall config test for "mutually exclusive application rule destinations" checks keys that don't exist in application rules (fqdn_targets).	That check never fails.	Review application rules by hand: use destination_fqdns or destination_fqdn_tags.
Sample exemption files (*.jason) use policy_definition_reference_ids. The module reads policy_definition_ids.	Renaming a sample to .json doesn't work as expected.	Use policy_definition_ids (Exemptions).
Management group JSON assignments must reference a policy set.	A single definition fails at plan time.	Wrap it in an initiative.
A custom initiative's Azure display name is its name (display_name = each.value.name).	properties.displayName is ignored.	Choose readable initiative names.
An undeclared variable in a tfvars file is only a Terraform warning, so typos and renamed variables are silently ignored.	The setting has no effect.	Check variable names against infra/deploy/<layer>/variables*.tf.

Open: Application Gateway and WAF

Issue	Impact	Workaround
policy_settings of WAF policies is in ignore_changes.	Mode or limit changes after creation are not deployed.	Change them in the portal and record it, or create a new policy.
create_application_gateway is overridden to true in the template.	Only use_application_gateway_module switches the gateway.	Use use_application_gateway_module.
The WAF template creates its own resource group (rg-<company>-agw-…) and ignores the gateway resource group passed in (rg-<company>-appgw-…).	WAF policies and the gateway live in different resource groups.	Build firewall_policy_id from the -agw- resource group.
In the gateway JSON, identity_ids is ignored (the template identity is always used), and kv_name, pip_name, uai_name and fips_enabled are not read.	Editing them has no effect.	–
Certificate passwords are read with azapi_resource .properties.value. With azapi 2.x this path may resolve to null. Not verified.	Password-protected PFX secrets may fail.	Store certificates without a separate password secret (Key Vault certificate import).

Operational pitfalls

These come from incidents and post-mortems on running platforms. They describe how the platform behaves during operation, not defects in one file. Check them against the version you pin.

Pitfall	Impact	What to do
A large firewall change (many rule collections added and removed at once) runs into the rule-set update limit: Azure does not accept concurrent updates of several rule sets, so the apply queues and ends in a time-out, and some rule collections exist in Azure but not in the state.	The next run fails with "already exists". Traffic can be interrupted while rules are missing.	Split the change into several smaller pull requests. Treat a time-out on canary as a reason to stop. Add new rules before removing old ones. Fix state differences with terraform import, not by destroying production rules (Post-mortems).
The firewall layer applies with -parallelism=1, so a big change takes a long time (Firewall rules).	A long run can exceed the apply time-out.	Keep changes small.
Enabling an extra WAF setting, such as inspection of the request body, without a test.	Legitimate requests match WAF rules and users get 403.	Test on canary. Record the change. See Application Gateway.
A missing private DNS record for a backend name.	The Application Gateway backend is unhealthy and users get 502. Easy to confuse with a firewall problem.	Check name resolution from the hub before you change firewall rules (Incident management).
A site-to-site VPN connection reports connected but passes no traffic.	Outage that no state-based alert catches.	Alert on traffic, not only on state. Use two connections where possible. Resetting the connection on both sides restored traffic in a reported case.
A destroy that fails halfway (for example, a stopped VM whose extensions cannot be removed) leaves a partial result.	A later re-run recreates parts, for example an empty data disk that is not attached.	Check what remains after a failed destroy, before you re-run or rebuild.
Resource or configuration names that contain the environment twice.	Wrong resource selected for deletion.	Compare the plan with the real resources and get the owner's confirmation before you apply a destroy.

Documentation that is out of date elsewhere in this repository

Document	Issue
docs/howto/run-initialize-pipeline.md, run-container-pipeline.md	They refer to initialize.yaml / stage-initialize.yaml and canary/production selector variables. The current pipeline is infrastructure.yaml, which reads everything from mcp.json.
PLATFORM_FEATURES.md	It says all resources get environment/managedBy/layer/region tags, and lists Terraform >= 1.0 / AzureRM >= 3.0. The code has no global tags (see Hub tags); the image pins Terraform 1.14.4, and the layers pin azurerm 4.60.0. For the resources and their switches, use What the platform deploys.
docs/policy_exemptions/current-platform-exemptions.md	Several lists differ from infra/deploy/postdeployment/main.tf, for example NIS2 and CIS 1.4.0 subscription exemptions.
docs/service_management/standard_changes.md	Older request forms. The request fields in this guide follow the current standard-change catalog.
CONTRIBUTING.md	It uses Resolve-XebiaMcpConfiguration -Environment (the function only has -Path) and refers to Functions/Public/Runner/, which doesn't exist.
docs/howto/configure-terraform-logging.md	It describes the terraformLogCore and terraformLogProvider parameters of the environment_canary and environment_prod pipelines. Check whether the entry pipelines you use expose them (Troubleshooting).
docs/howto/run-pool-smoke-tests.md	It calls Resolve-XebiaMcpConfiguration -Environment canary (the function only has -Path) and dev.ps1 -Action Initialize. Use the call shown in Troubleshooting.
Older update runbooks (single main.yaml pipeline)	They differ on the layer order (where ghrunners goes) and describe layers the current hub pipeline does not run. See Generic update runbook.
Release notes overview	It says the current stable version is v4.4.0, and ends at v4.6.1. See Release history.
Older alerting overview documents	Two documents list different severities for the same alerts (for example the VPN tunnel alerts), and one lists a firewall rule delete alert with an operation of another resource provider. Treat the alert definitions in the code as authoritative (Alert catalogue).