Serialize firewall layer apply for rule collection group updates¶

Context and Problem Statement¶

The firewall layer deploys Azure Firewall Policy Rule Collection Groups (RCGs) from a dynamic set of JSON files. Terraform applies resources in parallel by default. Azure Firewall Policy accepts only one RCG update at a time per policy. Concurrent writes return 409 AnotherOperationInProgress and can still fail after retries when queued updates run past provider timeout windows.

Decision Drivers¶

RCG file count is dynamic and not known up front.
Azure API behavior for firewall policy updates is serialized per policy.
Keep deployment throughput for non-firewall layers unchanged.
Avoid a high-risk refactor of the rule collection group module structure.
Prefer a targeted operational fix that works in both GitHub Actions and Azure DevOps.

Considered Options¶

Increase azurerm_firewall_policy_rule_collection_group timeouts only.
Refactor module/resource graph to force per-RCG sequential dependencies.
Run all Terraform apply operations with -parallelism=1.
Run only firewall layer apply with -parallelism=1.

Decision Outcome¶

Chosen option: "Run only firewall layer apply with -parallelism=1", because it matches Azure Firewall Policy concurrency limits and preserves deployment speed in other layers.

Consequences¶

Good, because sequential RCG updates avoid Azure 409 AnotherOperationInProgress conflicts.
Good, because only firewall apply changes; other layers keep normal parallelism.
Good, because the implementation is pipeline-local and low risk; no module refactor is required.
Neutral, because firewall apply takes longer when many RCGs are present.
Bad, because IP group operations in the firewall layer also run sequentially during apply.

Validation¶

This decision is validated by:

Firewall apply runs complete without 409 AnotherOperationInProgress during RCG updates.
Plan remains concurrent (no -parallelism=1), confirming only mutation operations are serialized.
Pipeline review confirms the flag is passed only for firewall apply in GitHub Actions and Azure DevOps.

Pros and Cons of the Options¶

Increase timeouts only¶

Good, because implementation effort is minimal.
Bad, because this does not prevent parallel conflicting writes.
Bad, because failures can still occur under sustained contention.

Refactor module/resource graph for sequential dependencies¶

Good, because sequencing could be expressed in Terraform configuration.
Good, because it could isolate sequencing to only RCG resources.
Bad, because RCG count is dynamic and Terraform does not support this pattern with dynamic per-instance depends_on chains.
Bad, because refactor complexity and migration risk are high.

Run all Terraform apply operations with `-parallelism=1`¶

Good, because it guarantees serialization and resolves firewall conflicts.
Bad, because it slows all layers and unrelated resources.

Run only firewall layer apply with `-parallelism=1`¶

Good, because it directly addresses the failing operation domain.
Good, because blast radius and performance impact are limited.
Neutral, because some safe-to-parallelize resources in firewall layer also run sequentially.