Reviewer and Operator Runbooks¶
This section holds the step-by-step runbooks of the people who handle standard changes: the operator who prepares and deploys a change, and the reviewer who checks a pull request. They are written for the platform team in both operating models: XMMS engineers in the managed-service model, your own engineers in the self-managed model.
The runbooks add the practical order of work to the technical pages of the configuration section:
| This section | The configuration section |
|---|---|
| What to check, in which order, and what to say when something is wrong | What the files look like and what each field does in Azure |
| How to deploy and how to verify the apply | Pipelines, layers and parameters |
| How to close the ticket | The request fields and the standard change catalog |
Keep both open: the runbook tells you the order, the common request tells you the details.
Runbooks¶
| Change | Operator runbook | Reviewer runbook | Deployment |
|---|---|---|---|
| Firewall rule | Handle a firewall rule request | Firewall rule pull request review | Firewall rule deployment |
| IP group | IP group change (operator and reviewer) | Firewall rule deployment (same layer) | |
| Application Gateway | Application Gateway change | WAF policy review for WAF policy files | Application Gateway deployment |
| Spoke | Spoke creation | Spoke pull request review | Spoke deployment |
| Managed VM | Managed VM pull request review (review, deployment and troubleshooting) |
The checks that apply to every pull request, the way to send one back and the auto-complete settings are on Pull request review. The process around the request is on The standard change process.
The shape of every standard change¶
Every runbook is a variation of the same sequence:
flowchart LR
a["1. Request is complete"] --> b["2. Branch and edit"]
b --> c["3. Validate and push"]
c --> d["4. Pull request and checks"]
d --> e["5. Review"]
e --> f["6. Deploy canary"]
f --> g["7. Verify the apply"]
g --> h["8. Deploy prod"]
h --> i["9. Verify and close the ticket"]| Step | Rule |
|---|---|
| 1. Request is complete | Do not start until all fields of the category are filled in. Ask the requester. |
| 2. Branch and edit | Branch from main. Change only the files the runbook names. Make the same change for every platform environment folder (canary and prod). |
| 3. Validate and push | Run the config tests locally if you can (Run the tests locally). |
| 4. Pull request and checks | Link the ticket. Wait for the validation pipeline. Fix what it reports. |
| 5. Review | A second engineer who did not write the change. See Pull request review. |
| 6. Deploy canary | Run the pipeline for the layer, for canary. Read the plan. |
| 7. Verify the apply | Open the apply log and check that the requested change was carried out. A green stage alone is not proof. |
| 8. Deploy prod | Same run for prod. The production plan is approved by a second engineer. |
| 9. Verify and close | Check the result as the runbook says. Add a line to the ticket, assign it to the requester and move it to Resolved. |
Rules that are the same everywhere¶
- Never change production first. Canary first, always.
- Do not trust a green pipeline alone. For complex resources, such as the Application Gateway, a successful apply stage does not show that the requested change is in place. Open the apply log. If the change is not visible, do not resolve the ticket: investigate, and ask a second engineer if needed.
- Stop on a destroy nobody expects. The plan of a standard change shows adds and in-place changes of the resources named in the runbook. Anything else is a reason to stop.
- Access. Before you start, make sure you have access to the tooling of the platform (the configuration repository, the pipelines and the ticket system) and to the right tenant. In the managed-service model an engineer who works for several customers must sign in with the account of the right customer.
- Do not copy secrets or personal data into tickets and pull requests.
- Feedback to the author. Send a pull request back with a comment on the ticket, a mention of the author, the ticket in state New and assigned to the author (Pull request review).