Skip to content

Subscription Billing Export

A cost report needs cost data of every subscription in one place. This page describes a pipeline that configures and starts a billing export for every subscription below a management group, and writes the data to a central data lake storage account. A reporting solution can then read it, for example a data factory pipeline that loads it into a cost dashboard.

For workload teams, the cost views in the portal are enough (Cost and reservations). This export is for central reporting by the platform team, FinOps or finance.

Origin of this page

The pipeline and script come from the spoke canvas of earlier versions. Names of files, repositories and variable groups are those of the starter files. Check what your code version ships.

What it does

Azure runs a subscription export at a moment it chooses, somewhere during the day. Exports of different subscriptions then arrive at different times, and the reporting shows stale or incomparable data. The pipeline does two things:

  • Configure. It scans all subscriptions below a management group (recursively) and creates the export definition in each.
  • Run. It starts the existing exports at a fixed time. The exports still do not finish at the same second, but within a much smaller window.

The export settings are:

Setting Value
Format Parquet, because it is compressed and works well with Azure Data Factory, the next step in the process
Compression Snappy
Granularity A daily summary
Overwrite behaviour Each day, the same file is written again with all data of the month
Time frame One month. In a new month a new file starts. 
flowchart LR
    p["Billing export pipeline<br/>(daily, fixed time)"] -->|"configure and start export"| s["Subscriptions below the<br/>landing zone management group"]
    s -->|"export to container billingdata"| d["Data lake storage account"]
    d --> r["Reporting solution<br/>(for example Data Factory, at a later time)"]

Prerequisites

Data lake storage account

  1. Create a storage account with hierarchical namespace enabled.
  2. Create a container, billingdata.
  3. Give the identity that runs the pipeline the role Storage Blob Data Contributor.
  4. Give the identity of the reporting pipeline (for example the managed identity of the data factory) the same role.

Script and pipeline

File Where Remark
BillingexportConfiguration.ps1 Scripts folder of the code repository Part of the platform. The pipeline checks it out.
BillingexportConfiguration.yaml pipelines folder of the configuration repository Starter file. Set its parameters.

Configure the pipeline file

Schedule. Run the pipeline before the reporting solution ingests the data. If the ingestion runs at 04:00, run the export in the evening, for example 22:00 local time. The schedule is in UTC, so convert the time (for example 0 20 * * * for 22:00 in a summer-time zone with UTC+2).

Parameters (the same names for the pipeline and the script):

Parameter Type Meaning Default or example
DoConfigureExports boolean Create or update the export definitions Pipeline: true. Script default: $false
DoRunExport boolean Start the existing exports Pipeline: true. Script default: $false
DoShowNextRunTimeEstimate boolean Show when the next automatic exports will run false in the pipeline, $true in the script
StartManagementGroupName string Highest management group from which to scan for subscriptions, recursively <landing-zone-management-group>
DataLakeStorageAccountResourceId string Resource ID of the data lake storage account /subscriptions/<subscription-id>/resourceGroups/<rg>/providers/Microsoft.Storage/storageAccounts/<name>
DataLakeStorageContainerName string Container for the data billingdata
DataLakeFolderNamePrefix string Folder prefix. The subscription ID is appended. subscription-
ExportJobNamePrefix string Prefix of the export name in the billing configuration exportprd_
SkipTheseSubscriptions string array Subscriptions that get no export @("<subscription-id>", "<subscription-id>")

The script also takes TenantId (the tenant that holds the management group).

Variable group. The pipeline uses the same variable group as the spoke deployment.

Code repository. Like the other pipelines of the spoke canvas, it checks out the code repository at a version tag:

resources:
  repositories:
    - repository: spokedeployment-tf
      type: git
      name: "<project>/<code-repository>"
      ref: refs/tags/<version>      # update when a new code version is deployed

Operate it

  • Run the pipeline by hand once, and authorise the pool and the group, like the other pipelines.
  • A new subscription gets its export at the next run of the pipeline with DoConfigureExports set to true. Data of a subscription is in a folder subscription-<subscription-id> of the container.
  • Check the run in the weekly pipeline status (Maintenance calendar). A failed run means a gap in the cost report.
  • A subscription that must not be exported goes in SkipTheseSubscriptions.