Use landing zone "feature flags" overwrites to temporarily remove expensive resources from canary environments¶
Context and Problem Statement¶
We deploy both canary and production environments for managed Azure landing zones at our customers. The canary environment is used to verify deployments and is nearly identical to production in terms of deployed resources and configuration. However, expensive resources like firewalls (€550-€2000/month) and application gateways (€250/month) accumulate significant costs in canary environments without being used for real workloads. Customers pay these bills, but it is our responsibility to ensure the Azure spend for resources we manage is appropriate and necessary.
Decision Drivers¶
- Cost responsibility - customers pay for unused expensive resources in canary environments
- Need for canary verification when customers request standard changes like firewall rules
- Requirement to balance cost savings with operational efficiency
- Preference to use lower SKUs in canary when possible without affecting resource deployment validation
Considered Options¶
- Scheduled destruction of entire canary landing zone
- Custom scripts to delete specific expensive resources
- Use Terraform variable overwrites to disable specific landing zone features
Decision Outcome¶
Chosen option: "Use Terraform variable overwrites to disable specific landing zone features", because this approach leverages existing Terraform feature flags like use_firewall_module=false, ensures proper dependency handling through Terraform's state management, and allows controlled recreation of resources when needed. The 20-30 minute recreation time for firewall and application gateway is acceptable when offset by the cost savings.
Consequences¶
- Good, because it reduces canary environment costs by €800-€2250/month per region
- Good, because it uses existing Terraform feature flags and infrastructure patterns
- Good, because Terraform handles resource dependencies correctly during both destruction and recreation
- Good, because it can be implemented with scheduled pipelines and template parameters
- Bad, because standard changes may take additional 20-30 minutes when resources need recreation
- Bad, because it adds operational complexity with additional pipelines and template parameters
Validation¶
This decision will be validated by:
- Testing the approach in our Demo customer environment first
- Monitoring the time impact on standard change validation
- Confirming cost savings match expectations
- Ensuring resource recreation works reliably when canary verification is needed
Pros and Cons of the Options¶
Scheduled destruction of entire canary landing zone¶
- Good, because it provides maximum cost savings
- Good, because it tests clean deployment scenarios regularly
- Bad, because full redeployment takes 3-4 hours for a two-region landing zone
- Bad, because it significantly delays standard change validation and implementation
Custom scripts to delete specific expensive resources¶
- Good, because it allows partial destruction of the landing zone
- Good, because it targets only the most expensive resources
- Bad, because resource dependencies must be hardcoded in scripts
- Bad, because scripts may break when Terraform structure changes
- Bad, because it doesn't leverage existing infrastructure patterns
Use Terraform variable overwrites to disable specific landing zone features¶
- Good, because it leverages existing feature flags like
use_firewall_module - Good, because Terraform handles resource dependencies correctly
- Good, because resources can be recreated reliably using the same mechanism
- Good, because it integrates cleanly with existing pipeline templates
- Neutral, because it requires pipeline modifications to accept additional Terraform arguments
- Bad, because standard changes may take additional 20-30 minutes when resources need recreation
- Bad, because it adds operational complexity with additional pipelines and template parameters
More Information¶
Initial implementation will focus on firewall and application gateway resources as they provide the highest cost savings with minimal risk to canary environment functionality. The approach will use scheduled pipelines with a "clean-up" template like:
template: .../clean-up.yml
inputs:
firewall: true
applicationGateway: true