Cost Insights Setup

This page is for the platform team. It describes how to enable the cost views (Insights) for an environment. It is a reference design: resource names, scopes and settings may differ in your deployment.

Data flow

flowchart LR
    be[Portal backend<br/>scheduled tasks] -->|creates exports| cm[Cost Management]
    cm -->|FOCUS Parquet, Snappy| st[(Storage account<br/>billingdata container)]
    cm -->|FOCUS Parquet, Snappy| dl[(Data Lake Gen2 accounts<br/>optional)]
    st --> adf[Data Factory<br/>daily pipeline, 04:00]
    adf --> sql[(Azure SQL<br/>tables and views)]
    sql --> be2[Portal backend<br/>queries at runtime]
    dl -.-> pg[(PostgreSQL<br/>azure_storage extension)]

The portal creates and maintains the exports itself. A Data Factory pipeline runs daily (trigger at 04:00 W. Europe time), loads the Parquet files into Azure SQL and refreshes the views that the portal queries.

Steps

Step	What
1	Add the cost_insights module to the environment root and wire its outputs into the backstage module
2	Add private connections for the SQL server, if the provider deploys from its own network
3	Set the cost export settings so the portal configures the exports
4	Deploy in two rounds and verify

1. The module

For each environment root, add the cost_insights module. Inputs come from outputs of existing modules (spoke canvas, Key Vault, naming and the backstage module).

module "cost_insights" {
  source = "../../modules/cost_insights"

  location                   = module.canvas.location
  environment                = module.canvas.environment
  workload_short_name        = module.canvas.workload_short_name
  subnets                    = module.canvas.subnets
  log_analytics_workspace_id = module.canvas.log_analytics_workspace_id
  key_vault_id               = module.key_vault.id
  data_factory_name          = module.naming.data_factory.name

  # One Data Lake Gen2 account per key. The key is part of the resource name,
  # so keep it stable: changing it recreates the account.
  datalake_storage_accounts = {
    default = {}
  }

  backstage_user_assigned_identity = {
    resource_name = module.backstage.user_assigned_identity.resource_name
    client_id     = module.backstage.user_assigned_identity.client_id
  }
}

The module creates a dedicated resource group with:

Resource	Purpose
Azure Data Factory	Loads the export files into SQL. Has a failure alert.
Storage account	Receives exports in the billingdata container and stores Data Factory audit logs
Data Lake Gen2 storage account	One per datalake_storage_accounts key, with its own exports
Azure SQL server and database	Processed billing data. The default SKU is S0 and the maximum size 150 GB.
User-assigned managed identities	Least-privilege access between Data Factory, storage and SQL
Key Vault keys	Customer-managed encryption for storage and SQL

An entry in datalake_storage_accounts can list readers: a map keyed by Entra object ID. Each reader gets Storage Blob Data Reader on that account, so another team, such as a data platform, can read the exported files. The map is empty by default, so nothing outside the portal can read the exports until you add a reader.

The module uses the mssql Terraform provider to manage database users, so the environment root needs this provider block. The runner that applies Terraform therefore needs a network path to the SQL server.

provider "mssql" {
  azure_auth = {}
  hostname   = module.cost_insights.database_server
  port       = module.cost_insights.database_port
}

Pass the outputs to the backstage module: the storage account resource IDs, the SQL server, port and database name (as app settings that start with APP_CONFIG_costInsights_database_), and the export name prefix.

module "backstage" {
  # ... existing configuration ...

  cost_exports_storage_account_resource_id = module.cost_insights.storage_account_resource_id
  cost_exports_datalake_storage_accounts = {
    default = {
      resource_id             = module.cost_insights.storage_account_datalake_resource_ids["default"]
      role_assignment_enabled = true
    }
  }

  additional_app_settings = {
    APP_CONFIG_costInsights_database_server   = module.cost_insights.database_server
    APP_CONFIG_costInsights_database_port     = module.cost_insights.database_port
    APP_CONFIG_costInsights_database_database = module.cost_insights.database_name
    APP_CONFIG_cost_exports_exportNamePrefix  = "<prefix>_"
  }
}

Use the module inputs, not the raw app settings

Setting the storage inputs of the backstage module is what creates the role assignments on each storage account. The equivalent app settings alone do not.

The portal loads the cost backend only when costInsights.database.server is set. Without it, the cost pages are not available.

2. Private connections

When the provider deploys from its own network, add a private connection for the SQL server (subresource sqlServer) in the platform root, then approve it on the customer side and register the DNS entry. See Onboarding a customer.

Cost exports configured by the portal

With the settings above, the cost backend runs scheduled tasks that create one FOCUS-schema Parquet export (Snappy) per in-scope subscription. Each export covers month-to-date data.

Task	Target
configure-azure-cost-exports	cost.exports: the storage account that Data Factory reads
configure-azure-cost-exports-datalake-<prefix>	One task per cost.dataLakeExports entry: its own Data Lake account

Behaviour	Detail
Batch	Each run handles a small batch of subscriptions (default 25)
Window	Runs only in an off-hours window and continues over runs until all subscriptions are configured
Existing exports	Detected and left in place
Reprocess interval	One day by default
Dry run	A dryRun setting logs what would change without creating exports
Prefix	The export name prefix defaults to the storage account name plus _
State	Each target tracks its own state, so tasks do not interfere

Scope the subscriptions when "every subscription the identity can see" is too broad:

cost:
  exports:
    includeManagementGroups: ['<management-group>']
  dataLakeExports:
    default:
      includeManagementGroups: ['<management-group>']

Requirements on the Azure side:

• The portal's managed identity needs Cost Management Contributor on the in-scope management group and read access to both storage accounts (Prerequisites).
• Microsoft.CostManagement must be registered on each in-scope subscription. If it is not, the portal logs a warning and tries again on the next run.

Direct ingestion into PostgreSQL

The portal can import the Parquet files from the Data Lake accounts into its own PostgreSQL database. The postgresql module prepares the server: it has a system-assigned managed identity, the backstage module gives it Storage Blob Data Reader on each Data Lake account, and the azure_storage extension is allowed and preloaded. Apply the change that creates the identity before the change that assigns the role, because Terraform cannot plan a role assignment for an unknown principal. Changing shared_preload_libraries restarts the server.

The source describes this path as preparation for a future phase in which the portal ingests cost data itself and combines it with catalog data.

Migrating from an earlier export pipeline

If a customer still runs an earlier, script-based export pipeline:

1. Deploy this version with the same export name prefix that the pipeline uses per environment, so both target the same export.
2. Check the backend logs for the configure-azure-cost-exports task and confirm that the exports exist in the Azure portal.
3. Disable and delete the old pipeline.

Deploy and verify

1. Deploy the first environment. The first run skips creation of database tables and privileges because the Data Factory resource group does not exist yet, so run a second deployment.
2. Approve the private endpoint connections and update DNS.
3. Deploy the next environment in the same way.
4. Confirm that the portal has configured the exports and that at least one file has landed in the billingdata container. Exports usually run within 24 hours and ingestion overnight, so data can take one to two days to appear.

Related

• Insights
• Operations