Skip to content

Testing the Templates

Every module is tested for two things: that it works (it deploys, and the resources behave as expected) and that it is compliant (the resources it creates do not violate the Azure policies of the platform). The tests run in a pipeline, in parallel, in a dedicated test subscription.

The test process

flowchart TD
    a1["Terraform: Terratest"] --> a2["Integration test: deploy the example"]
    b1["Bicep: Pester"] --> b2["Integration test: deploy the example"]
    a2 --> c["Azure Policy scan of the resource group"]
    b2 --> c
    c --> d{"Any non-compliant policy?"}
    d -->|yes| e["Test fails"]
    d -->|no| f["Test passes"]
    f --> g["Resources are removed. Shared resources stay."]
  1. The test deploys the example of a module (examples/<technology>/<module>) into its own resource group.
  2. The tool of the technology checks that the deployment works.
  3. A PowerShell script triggers a policy evaluation on the resource group, reads the compliance state of the policies that are assigned in the test subscription, and fails the test when any policy is non-compliant.
  4. The resources of the test are removed. The shared resources stay (below).

Compliance with Azure Policy

The templates must meet the baselines of the platform: the CIS Microsoft Azure Foundations Benchmark 2.0, ISO 27001:2013 and the Microsoft cloud security benchmark. Instead of writing and maintaining a second set of rules in a code scanner, the tests use Azure Policy as the single source of truth:

  • The policies of the platform are assigned at the level of the test subscription.
  • Each template is deployed in its own resource group, so that the result belongs to one template.
  • After the deployment, the test starts a policy evaluation and then reads the compliance result.
Good Independent of the language (Bicep or Terraform). The policies are the source of truth. Little to maintain. Simple to implement.
Bad Slower than a static scan. The test deploys the resources, evaluates the policies and removes the resources, which takes minutes per module.

Code scanners (static analysis of Terraform and Bicep) were tried and rejected for now: they gave false positives on nested modules and on loops, missed some checks, and had poor Bicep support. They can be added later, if the test time becomes a problem for development.

The policy check

The script tests/Get-PolicyResult.ps1 does the check for both technologies. It takes the name of the resource group and an optional list of policy reference IDs to ignore:

  1. It checks that the resource group exists, and starts a policy scan on it (az policy state trigger-scan).
  2. It lists the policy states with the state NonCompliant, with the assignment name and the policy reference ID.
  3. It removes the ignored policies from the count, and returns the number of non-compliant policies.

The test passes only when the number is 0.

Ignored policies

A test can pass a list of policy reference IDs to ignore. The list is kept short, and every entry has a comment that gives the reason in the test file. The reasons are in two groups:

  • The policy cannot be evaluated or fixed by the module: for example the flow-log policy for network security groups (flow logs are no longer possible on that level), the vulnerability assessment policies for registries and databases (they need time or a subscription plan), and the policies for unused or unauthenticated APIs on the API Management Developer SKU (Azure creates an Echo API).
  • The policy needs more time than a test can wait: guest configuration policies on virtual machines are evaluated later and the test cannot trigger them.

The same findings are documented for consumers in Known findings. Adding an ignored policy is a decision to review: explain it in the test file and in the module notes.

Tools per technology

Technology Tool Why
Terraform Terratest (a Go library) Mature and widely used. Runs tests in parallel. Flexible, and can be used for other tests. Needs a little knowledge of Go to build the first test structure.
Bicep Pester (PowerShell test framework) Little effort to capture the output and create a test report that Azure DevOps can read. A script runs the tests in parallel, because Pester has no native parallelism. Known weaknesses: the total test time in the report is not correct, and the log shows output only when all tests are done.

Two options were not chosen: the native terraform test framework (no parallelism, less flexible and less mature) and a custom PowerShell loop with parallel deployments for Bicep (flexible, but a lot of work to build and maintain).

How a test is built

  • Terraform: one Go test per module in tests/terraform/<module>_test.go, named Test<ModuleName>. It runs in the stages apply, policy validation and destroy. It uses the backend of the test state storage with a key based on the path of the example, so each example has its own state. Common helpers are in common.go.
  • Bicep: one Pester file per module in tests/bicep, with the tests deploys and validates policies, and a cleanup that deletes the resource group. A runner script starts all test files in parallel (one thread job per file), merges the results, and writes a JUnit report.

The scaffolding script creates the Terraform test from a template when you create a module (Pipelines and scripts). For Bicep, copy an existing test file and adapt it.

Shared resources during tests

Most modules need the same dependencies: a virtual network and subnet, a Log Analytics workspace, a Key Vault, a storage account. Deploying all of them for every test would make the tests slow and the examples complex.

Instead, the dependencies are created once, and every test reuses them:

  1. The shared resources are defined in Terraform in the dependencies/ folder. The storage account for the Terraform state is the one resource that is not created from there: a small Bicep template creates it.
  2. A pipeline creates them in its first phase, if they do not exist yet.
  3. The pipeline returns information about the shared resources as a JSON object, and passes it to the following jobs.
  4. Before a module is deployed, a job writes the JSON to a file named shared_resources.json. The Terraform example reads it with jsondecode(file(...)), the Bicep example with loadJsonContent('shared_resources.json').
Good Short tests. The dependencies are not tested in every integration test: the module is. Terraform and Bicep tests reuse the same resources. All dependencies are changed in one place. Fewer causes for a failed test.
Bad More complex. The editor cannot autocomplete the values of the shared resources.

The alternative (every module brings its own dependencies) keeps modules separate, but creates many copies of the same resources, makes unique names a problem, and takes a lot longer.

What the shared resources are

Shared resource Used for
Resource group, virtual network and subnet, with a network security group Private endpoints and VM network interfaces. The network is peered with the network of the build agents, so that the agents can reach private resources.
Private DNS zones for the common private link services (blob, table, queue, file, PostgreSQL, container registry, Key Vault, SQL, Service Bus, App Configuration), linked to the network of the agents Name resolution for the private endpoints of the examples
User-assigned identity, with the roles it needs on the Key Vault Customer-managed keys and other identity tests
Key Vault, Log Analytics workspace, storage account, container registry Dependencies of many modules
Recovery Services vault with a daily VM backup policy Virtual machine tests
A generated SSH key pair, stored in the Key Vault Linux virtual machine tests

The identities and the network of the agents are the ones of the maintaining team. Your own test environment needs its own.

Run the tests

  • Normally: use the CI pipeline of the technology. It runs on pull requests that change that technology, and every week (Pipelines and scripts). A pull request runs the tests of the modules it changes, and all tests when it changes files outside the module, example and test folders. After the run, everything created during the tests is removed, except the shared resources.
  • One test: start the CI pipeline by hand and give the name of the test (Terraform) or of the test file (Bicep).
  • Manually: a pipeline can run apply or destroy for one path of the repository (for example examples/bicep/<module>), with an option to run the policy scan afterwards. Use it to try a template while you build it. It runs on private build agents.
  • Locally: the development container creates the shared resources with Terraform and writes shared_resources.json for both test folders, so that you can run a test from your own machine against a test subscription.