AKS Platform: Load Testing¶
This page describes how the platform is load tested and what the test measures. The test has two goals: find out how much traffic a workload on the non-production cluster can take before it breaks, and prove that the alerting chain (Prometheus rule, Alertmanager, PagerDuty) fires and clears when it should.
Reference design, your deployment may differ
The test runs against the non-production cluster only. The target application, thresholds and routes are those of the reference setup. A longer design document describes a wider test programme with another tool. This page covers what is implemented and lists the rest as planned.
How a test run works¶
A manually started pipeline creates a short-lived job in the cluster, drives traffic in three stages, watches the effects and removes everything again. The cluster carries no load generator when idle.
flowchart TB
start["Pipeline started by hand<br/>(rates, durations, failure trigger)"] --> login["OIDC login to Azure,<br/>cluster credentials"]
login --> infra["Apply namespace, RBAC,<br/>network policy, dashboard"]
infra --> mark1["Annotate Grafana: test start"]
mark1 --> job["Vegeta job in the load-testing namespace"]
job --> s1["Stage 1: warm-up"]
s1 --> s2["Stage 2: sustained"]
s2 --> s3["Stage 3: overload"]
s3 --> report["Vegeta report in the run summary"]
report --> mark2["Annotate Grafana: test end"]
mark2 --> clean["Delete job and config map,<br/>reset the failure rate"]| Part | Detail |
|---|---|
| Tool | Vegeta, a constant-rate HTTP attacker, run as a Kubernetes job (one pod, no retries) |
| Where | Inside the cluster in a dedicated load-testing namespace, so traffic goes through the cluster network and DNS |
| Trigger | A manual GitHub Actions workflow with the parameters below |
| Identity | The pipeline's federated identity with the Azure role Azure Kubernetes Service RBAC Admin, scoped to the cluster |
| Footprint | A service account with a namespaced role, a network policy for the generator pods, and a Grafana dashboard config map. The job and its script config map are deleted at the end (if: always()); the job also has a time-to-live |
| Generator size | Requests 500m CPU and 1 GiB, limits 2 CPU and 4 GiB. The worker pool is capped at 8000 connections, which is needed to hold 3000 requests per second when latency rises |
| Pre-flight | The script waits up to 60 seconds for the target's health endpoint before it attacks |
Stages and parameters¶
| Stage | Default rate | Default duration | Purpose |
|---|---|---|---|
| 1. Warm-up | 60 requests per second | 30 s | Healthy baseline |
| 2. Sustained | 400 requests per second | 1 min | Load that makes the autoscaler react |
| 3. Overload | 3000 requests per second | 5 min | Push past the capacity of the workload |
All six values are workflow inputs. A seventh input, trigger_failure (default on), automates the failure story: the pipeline
raises the application's failure rate at the start of stage 2 and resets it in the last minute of stage 3, so an error spike is
guaranteed without hand-tuning rates. Without it you drive the two changes by hand with kubectl set env.
What is measured¶
The application exports its own Prometheus metrics (the generator itself exports none). The run is judged on the application's and the cluster's side.
| Area | Signal | Used for |
|---|---|---|
| Request rate | Requests per second by response code | Offered load and achieved throughput |
| Errors | Ratio of 5xx responses over 5 minutes | Availability and the fast-burn alert |
| Latency | p50, p90, p95, p99 from the request duration histogram | Latency objective |
| Autoscaling | HPA desired replicas against current replicas | Does the workload scale out, and how fast |
| Capacity | Pod CPU against requests, pending pods, restarts, out-of-memory kills | Where the workload saturates |
| Alerts | Prometheus alert state with the load test label, Alertmanager receiver, PagerDuty incident | Whether the chain works |
| Client view | The Vegeta report: totals per stage and combined, with latency percentiles and success ratio | Cross-check from the client side |
A Grafana dashboard for the test is provisioned from a config map. A helper script can print a summary from Prometheus after the run (total requests, peak rate, peak error rate, peak p95 and p99, maximum replicas, restarts, alert state).
Pass and fail¶
The job does not pass or fail on Vegeta's success ratio, because stage 3 is meant to overload the workload. The workflow fails only if the job errors (for example the target is unreachable) or does not finish in time. The real result is whether the alerts fire and resolve and an incident is created and cleared.
Alert validation¶
Three load test alert rules mirror the service level objectives of the reference assignment. They carry labels that mark them as test alerts:
| Alert | Condition | For | Severity |
|---|---|---|---|
AvailabilityErrorBudgetBurnFast |
5xx ratio over 5 minutes above 10 times the 0.1% error budget | 5 min | page |
AvailabilityErrorBudgetBurnSlow |
5xx ratio over 30 minutes above 2 times the budget | 12 h | ticket |
LatencySLOViolation |
p95 latency over 5 minutes above 300 ms | 15 min | page |
Alertmanager routes alerts with source = load-test to a dedicated receiver. In the implemented setup it uses the same
PagerDuty routing key as production and is told apart by a [LOAD-TEST] prefix and fields in the event. The design document
proposed a separate PagerDuty service for this.
What you can observe in one run: only the fast-burn alert, because the default stage 3 lasts 5 minutes, as long as the rule's
for period. Run stage 3 for 20 minutes or more to see the latency alert fire. The slow-burn alert cannot be observed live.
Evidence to collect for an alert validation:
| Check | How |
|---|---|
Alert reaches firing |
Prometheus alerts page, filtered on the test label |
| Routed to the test receiver | Alertmanager alerts page |
| Incident created | PagerDuty, title with the [LOAD-TEST] prefix |
| Incident resolves by itself | PagerDuty, after the failure rate is reset and the metrics recover |
| No effect on production alerting | No other incidents on the same service in the window |
The target workload¶
The target is a small public demo web application (Flask, with Prometheus metrics) deployed in the <team>-dev namespace with
the shared application chart: autoscaling from 3 to 10 replicas on a 70% CPU target, a disruption budget, a hardened security
context, and a service monitor that sets the Prometheus job label so that rules, dashboard and queries all key on one name.
The work endpoint simulates 50 to 200 ms of latency and fails randomly at a configurable rate (default 2%). It degrades
realistically under load without any artificial stress switch.
Only the dev environment is supported. The test and accp namespaces run a placeholder web server image, not the demo
application.
What the numbers mean
In the validated run, throughput levelled off at about 1,800 to 2,260 requests per second (about 220 per pod at ten pods), the autoscaler went from 3 to 10 pods, the fast-burn alert fired about six minutes into the run and resolved shortly after the failure rate was reset. These numbers describe the demo application (a single-process development server), not the capacity of the platform. Raising the CPU limit made no difference. Treat the run as a repeatable method for finding the number for any service.
Known limitations¶
| Limitation | Effect |
|---|---|
| The latency histogram measures only request handling, not time spent queued when the server is saturated | The latency alert rarely fires even under severe overload, so the fast-burn alert is the reliable signal |
| An Argo CD self-heal reverted the failure-rate change within a second | Fixed by telling the Argo CD application to ignore that one field |
| Role assignments for the pipeline and the Graph API issue | Use the cicd-aks-rbac target (Deployment) |
| Results of runs are not stored or compared automatically | Capture the run summary and the dashboard yourself |
| Local runs on a laptop give noisy absolute numbers | Use in-cluster numbers |
Planned test programme (design only)¶
The design document describes further test types for the non-production cluster. They are not implemented in the pipeline.
| Test | Pattern | Purpose |
|---|---|---|
| Smoke | 1 to 5 users, 1 to 2 minutes | Check the test setup and the target |
| Baseline | Normal load for about 13 minutes | Reference values for release comparison |
| Load | Peak expected load for about 23 minutes | All platform objectives hold |
| Stress | Step-wise ramp until degradation | Find the breaking point, informational |
| Spike | Sudden burst, then back | The autoscaler reacts and the system recovers |
| Soak | Moderate load for 2 to 8 hours | Leaks and gradual degradation |
| Capacity | Ramp until one objective breaks | Maximum sustainable throughput |
| Autoscaling | Load above the autoscaler target, then to zero | Scale-up and scale-down timing |
| Alerting | Load patterns that trigger known alerts | Rule to incident chain |
The design also named k6 with its Kubernetes operator as the primary tool and an Azure DevOps pipeline. The implementation moved to Vegeta, driven from GitHub Actions, because the k6-based demo runs repeatedly failed to give the expected results.