Observability (Managed Grafana)¶

This runbook is for the platform team. It covers the optional observability tier: an Azure Managed Grafana instance per environment that reads Log Analytics and shows the health of Fabric pipelines. Workload teams read Monitoring.

What it deploys¶

The tier is a separate Terraform root (infra/deploy/observability) with its own state (observability.tfstate), so it has its own lifecycle. The only dependency is the Log Analytics workspace of the spoke, which must exist first.

flowchart LR
    fp["Fabric pipelines"] --> lh["pipeline_logs Lakehouse"]
    lh --> law["Log Analytics workspace"]
    law --> g["Managed Grafana (KQL dashboards)"]
    law --> alert["Scheduled query rule: pipeline failures"]
    law -->|"data export rule (optional)"| st["Storage account: cool, then archive, then delete"]

Resource	Notes
Resource group	`rg-<workload>-observability-<env>-<region>-001`
Managed Grafana	`grafana-<workload>-<env>-<region>`, SKU `Standard`, size `X1` by default, major version `12`, system-assigned identity, deterministic outbound IPs on, API keys off, public access on by default, zone redundancy off by default
Role assignments for the Grafana identity	Monitoring Reader and Log Analytics Reader on each connected Log Analytics workspace
Role assignments for groups	Grafana Admin, Grafana Editor and Grafana Viewer for the groups in `grafana_rbac`
Private endpoint (optional)	In the spoke private endpoint subnet. Private DNS is set by policy, so the Terraform ignores changes to the DNS zone group.
Diagnostic setting	Grafana audit logs and metrics to the first Log Analytics workspace
Alert rule and action group	`alert-pipeline-failure-grafana-...` and `ag-pipeline-alerts-grafana-...`
Historical log storage (optional)	A storage account with a data export rule and a lifecycle policy

Design decisions¶

Decision	Reason
Dedicated resource group and state	Isolates observability from workload resources and allows an independent lifecycle
System-assigned managed identity	Grafana reads Log Analytics without credentials, over the Azure backbone
Deterministic outbound IPs	Egress addresses are fixed, so they can be identified in logs and allowed in firewalls
Entra ID only, no API keys	All access is by Entra ID with MFA through Conditional Access. Automation uses a service principal or managed identity.
Separate groups for Grafana	Different blast radius: Grafana access is not Fabric access
Public by default, private endpoint optional	Development uses public access with Entra ID and MFA. Production can restrict to the VNet.
Dashboards as code	JSON templates in the configuration repository, provisioned with the Grafana Terraform provider

Prerequisites¶

Security groups¶

Create three Entra ID security groups per workload before deploying. They are separate from the Fabric workspace groups.

Group (naming pattern)	Grafana role	Intended members
`sg-fabric-<workload>-grafana-admins`	Admin	Platform team engineers
`sg-fabric-<workload>-grafana-editors`	Editor	Data engineers and analysts of the workload team
`sg-fabric-<workload>-grafana-viewers`	Viewer	Business stakeholders, management

Creating groups needs the Groups Administrator or Global Administrator role in Entra ID.

WORKLOAD="<workload>"
for role in admins editors viewers; do
  az ad group create \
    --display-name "sg-fabric-${WORKLOAD}-grafana-${role}" \
    --mail-nickname "sg-fabric-${WORKLOAD}-grafana-${role}" \
    --security-enabled true --query id -o tsv
done

Add members with az ad group member add --group <name> --member-id <object id>. Put the object IDs in the variables:

grafana_rbac = {
  admins  = ["<admins group object id>", "<deploying service principal object id>"]
  editors = ["<editors group object id>"]
  viewers = ["<viewers group object id>"]
}

The deploying service principal must be Grafana Admin

The dashboards are provisioned through the Grafana API. That needs the Grafana Admin role on the instance, which is a separate assignment from the Contributor rights that create the instance. Put the object ID of the deploying service principal in grafana_rbac.admins.

Other¶

The Log Analytics workspace IDs to connect (log_analytics_workspace_ids). The first one is used for diagnostics, the alert rule, the data export rule and the dashboard template variable.
Optionally the subnet and private DNS zone for a private endpoint.

Variables¶

Variable	Purpose and default
`grafana_config`	`sku` (`Standard`), `sku_size` (`X1`), `api_key_enabled` (`false`), `deterministic_outbound_ip_enabled` (`true`), `public_network_access` (`true`), `zone_redundancy` (`false`), `grafana_major_version` (`"12"`)
`grafana_private_endpoint`	`enabled` (`false`), `subnet_id`, `private_dns_zone_id`
`grafana_rbac`	Lists of object IDs for `admins`, `editors`, `viewers`
`alert_rules`	`pipeline_failure_enabled` (`false`), `pipeline_failure_interval` (300 seconds), `pipeline_failure_lookback` (3600 seconds), `notification_contact_emails`
`dashboard_definitions`	`enabled` (`false`)
`dashboard_path`	Directory with the dashboards. Set by the pipeline through the environment variable `TF_VAR_dashboard_path`.
`historical_log_storage`	See Historical log retention

Allowed regions and environments are the same as for the other tiers (Platform configuration).

Deployment¶

The pipeline has a run option to deploy the observability tier. The pipeline definitions are not part of this solution's source, so the exact option name depends on your pipeline.

Two phases on the first deployment. The Grafana provider needs the URL of an instance that does not exist yet.

Phase	What happens
First run	The Grafana instance, role assignments, diagnostics and alert rule are created. Dashboard resources fail because there is no endpoint yet. This is expected.
Second run	The pipeline reads the `grafana_endpoint` output, gets a token for the Azure Managed Grafana application (`GRAFANA_URL` and `GRAFANA_AUTH` environment variables) and the dashboards are provisioned

After that, every run is a single phase. Data engineers add dashboards in the configuration repository and set dashboard_definitions.enabled = true.

The dashboards directory has one sub-folder per Grafana folder, and one .json file per dashboard. The module discovers all */*.json files, so no per-file configuration is needed. The file is a Terraform template with one variable: ${LAW_RESOURCE_ID}. Grafana template variables must be escaped ($${...}).

Security controls¶

Control	Implementation
Authentication	Entra ID SSO, no local accounts
MFA	Conditional Access policies
API keys	Disabled by default. If enabled, rotation is manual.
Audit logging	Diagnostic settings to Log Analytics (audit logs, all logs, metrics)
Outbound IPs	Deterministic
TLS	HTTPS only
DNS label scope	`TenantReuse`, which gives a predictable DNS name
Private DNS	Managed by policy

Historical log retention¶

Logs can be tiered for cheaper long-term retention. It is off by default (historical_log_storage.enabled = false).

Tier	Duration	Storage	Query
Hot	0 to 30 days	Log Analytics workspace	Interactive KQL
Cool	31 to 90 days	Blob storage, cool	Log search jobs or restore
Archive	91 to 365 days	Blob storage, archive	Rehydrate first (takes hours)

Setting	Default
`export_tables`	`["FabricPipelineRuns_CL"]`
`archive_after_days`	`90`
`delete_after_days`	`365`
`law_resource_group_name`	Resource group of the Log Analytics workspace, needed for the export rule

The storage account is Standard, LRS, StorageV2, access tier Cool, with blob and container soft delete of 7 days. A lifecycle rule moves blobs to archive and deletes them after the configured days. Prerequisites: the custom table must exist in Log Analytics before the export rule can be created, and export rules only support DCR-based custom tables (suffix _CL). Deploy historical storage after the data engineers have set up the log ingestion pipeline.

Troubleshooting¶

Symptom	Cause	Fix
"No data" in all panels	The `FabricPipelineRuns` table does not exist yet	The data engineering pipeline has not filled Log Analytics yet
"No data" in some panels	Time range too narrow, or no runs of that status	Widen the time range, check that pipelines ran
"Insufficient permissions" on the data source	Role assignments of the Grafana identity have not propagated	Wait 10 minutes after deployment
Cannot sign in (403)	User is not in a Grafana group	Add the user to a group
Cannot sign in (timeout) in private mode	VPN or ExpressRoute is not connected, or DNS does not return a private IP	Connect and check DNS
"Datasource not found"	The Azure Monitor data source is not configured	Check that the Grafana identity has Monitoring Reader on the workspace
Alert does not fire	Rule disabled, or no email configured	Check `pipeline_failure_enabled` and `notification_contact_emails`
Deployment fails on `grafana_major_version`	The Standard SKU needs a supported major version	Set `grafana_major_version = "12"`
Terraform wants to change `private_dns_zone_group`	Policy set the DNS zone group after creation	Expected. Keep the `ignore_changes` lifecycle block.

Known limitations¶

No folder-level or dashboard-level permissions in Azure Managed Grafana. Options if you need separation: one instance per team (each with its own groups and Log Analytics connections), accept a shared instance, or run your own Grafana Enterprise (more cost and operations).
No automated rotation of Grafana API keys, if you enable them.
Pipeline failure alerts send email only when notification_contact_emails is filled.
Only the first Log Analytics workspace is used for the alert rule and the dashboard variable.