Pipelines and Scripts of the Library¶

The repository automates testing, documentation, release and delivery. This page lists the pipelines in pipelines/ and the scripts in scripts/, so that the maintaining team knows what runs when. The release flow itself is in Release and documentation, the tests in Testing.

Reference design, your deployment may differ

The pipelines are Azure Pipelines definitions that run on private agents and use a service connection to a test subscription. Names of pools, service connections and subscriptions are specific to the maintaining team and are not listed here.

Pipelines¶

Pipeline	Trigger	Purpose
CI for Terraform (`CI-terraform.yaml`)	Pull requests that change `modules/terraform`, `examples/terraform` or `tests/terraform`, and a weekly schedule	Generates the module documentation, creates the shared resources, runs the Terratest tests
CI for Bicep (`CI-bicep.yaml`)	Pull requests that change `modules/bicep`, `examples/bicep` or `tests/bicep`, and a weekly schedule	The same for Bicep, with Pester
Release (`release.yaml`)	Every push to `main`	Creates the new version tags per module
Manual deploy and destroy (`deploy_destroy_manual.yaml`)	Manual	Deploys or removes one example, to build and try a module
IaC accelerator for Terraform and for Bicep (`terraform_iac_accelerator.yaml`, `bicep_iac_accelerator.yaml`)	Manual	Deploys the accelerator project to the dev or prod test subscription

The pipelines use four shared templates in pipelines/templates/: one to create the shared resources (dependencies.yaml), one to generate and commit documentation (docs.yaml), and one each to deploy Terraform and Bicep examples (terraform.yaml, bicep.yaml).

CI pipelines¶

Both CI pipelines have the same three jobs:

flowchart LR
    docs["docs: generate and commit documentation"] --> shared["shared_resources: create dependencies"]
    shared --> deploy["deployment: run the tests"]

Job	What it does
docs	Runs on every branch except `main`. Updates the template index in the main README, regenerates the module documentation (`terraform-docs` for Terraform, the `Set-Module` script for Bicep), and commits the changed `README.md` and `main.json` files back to the branch with the message Updated documentation [skip ci].
shared_resources	Creates the Terraform state storage with a Bicep template, runs `terraform apply` for `dependencies/`, and exports the `shared_resources` output as JSON for the next job (Shared resources).
deployment	Writes the JSON to `shared_resources.json`, runs the tests, converts the results to a JUnit report (the Terraform log parser of Terratest, the Pester JUnit export) and publishes the test results. The job has a timeout of 180 minutes.

The tests run in parallel: Terratest with -parallel 10, Pester with one thread job per test file.

What is tested. A scheduled run and a manual run test everything. A pull request runs only the tests of the modules it changes. It finds them from the changed files in modules, examples and tests. When the pull request changes anything else (for example a pipeline or a script), all tests run. Changes to Markdown files and to iac_accelerator/ do not count as such changes. The weekly run (Tuesday, early morning, UTC) always runs on main, also when nothing changed.

Parameters for a manual run.

Pipeline	Parameter	Meaning
CI for Terraform	`SpecificTest`	The name of one test (for example `TestKeyVault`). The default `*` runs all tests.
CI for Terraform	`DocsOnly`	Generate only the documentation and skip the tests
CI for Bicep	`SpecificTest`	A test file filter (for example `keyVault.Tests.ps1`). The default runs all.
CI for Bicep	`SkipTests`, `SkipDocs`	Skip the tests or the documentation
CI for Bicep	`moduleName`	The module to generate documentation for, or `all`

Release pipeline¶

The release pipeline runs for every push to main and not for pull requests. It checks out the repository with two commits of history and runs scripts/New-Release.ps1, which goes through all module folders and creates a tag for each module that changed. See Release and documentation and Versioning.

The pipeline does not run tests. The tests run on the pull request and every week.

Manual deploy and destroy¶

Parameter	Meaning
`action`	`apply` or `destroy`
`template_path`	The path to deploy from. It must contain `terraform` or `bicep`. The default is the Terraform examples folder.
`enable_policy_scan`	Run the policy check after the deployment

The pipeline first creates the shared resources, then deploys the example. For Terraform it runs init (with the state storage as backend), validate, plan and apply or destroy. For Bicep it runs az deployment sub create, and for destroy it deletes the resource group that the deployment returned.

Scripts¶

All scripts are PowerShell.

Script	Purpose
`New-Release.ps1`	Goes through every module folder and calls `New-ModuleRelease` for it. Empty folders are skipped.
`New-ModuleRelease.ps1`	For one module: checks if the module qualifies for a release, calculates the target version and creates and pushes the tag.
`Set-Module.ps1`	Generates what can be generated for a Bicep module: the compiled `main.json`, the `README.md`, and missing files and folders. Options skip each of the three, and `-Recurse` handles all modules in a path.
`New-TerraformModule.ps1`	Scaffolds a new Terraform module: the folder, `_header.md`, `CHANGELOG.md`, `version.json`, `main.tf`, empty variable and output files, the docs example, a Terratest test and an example, with the module name filled in.
`Set-XmtIndex.ps1`	Regenerates the template index in the main README: a table with every module and a link for each technology that has it.
`Publish-ModuleToACR.ps1`	Publishes the Bicep modules of a module to a container registry, for one version or all versions, from the git tags.
`helper/`	Functions that the scripts above use: version calculation, change detection, tag creation, README generation for Bicep, and checks for referenced modules. Templates for new modules are in `helper/module_templates`.

Publish-ModuleToACR.ps1 checks that the tag exists, that the module has a main.bicep, and that the registry is reachable with push rights. It skips versions that the registry already has, and publishes the others as modules/<technology>/<module>:<version>, with a link to the README at that tag as documentation. The pipeline that calls it for each consumer is kept outside this repository (Release and documentation).

Version calculation in short¶

The scripts compare the files of the last commit with the filter in version.json to decide whether a module changed. Then they build the version from the MAJOR.MINOR of version.json and a calculated PATCH: reset to 0 when version.json changed, otherwise the highest existing patch plus one. Details are in Versioning.

Developer environment¶

The .devcontainer/ folder defines a development container with Terraform, terraform-docs, Azure CLI with Bicep, PowerShell, Go, Git and Docker-in-Docker. Its setup script logs in with a service principal, creates the shared resources and writes shared_resources.json for the Terraform and Bicep tests, and installs Pester. Contributors use it to run tests locally.

Automated updates of the accelerator¶

A scheduled workflow runs Renovate every week on the repository, limited to iac_accelerator/. It keeps the module versions and providers in the accelerator current, and opens a pull request per update group. Consumers get the same setup in their copy of the accelerator (Keep modules up to date).