Skip to content

Monitoring

Monitoring is split over two layers: the platform team monitors the platform components, the workload team monitors its own workloads. Optional Grafana dashboards give both a view on pipeline health.

Reference design, your deployment may differ

Who receives alerts, and which optional parts (Grafana, Purview) are deployed, is set per platform. Ask your platform team. Generic platform monitoring is described in Monitoring.

Platform monitoring (platform team)

Component What is watched
Fabric capacity Capacity unit consumption, throttling events and utilisation, through the Fabric admin portal. This lets the platform team scale up before the load reaches the capacity.
VNet data gateway Availability, connection counts and latency. The gateway sleeps after inactivity and wakes on a new request, so the activation cycle matters when judging connection performance.
Storage account Availability, latency and capacity usage
Connections Availability of the ADLS, SQL, Oracle Cloud Storage and workspace-to-workspace connections
Alerts Deviations are signalled to administrators. Diagnostic and log data are stored centrally for analysis.

Baseline pipelines

The platform team monitors the baseline pipelines of the platform. If a pipeline fails, the platform team retries it. The cause is usually the source and a retry solves it most of the time. If the retry does not help, the platform team reports it to the team that owns the pipeline.

Workload monitoring (workload team)

Area What you do
Pipeline execution Data pipelines produce run status, duration and error messages. When a retry by the platform team does not solve the failure, the workload data team solves the data issue.
Query performance Durations, resource consumption and timeouts on Lakehouses and Warehouses
Data quality and freshness Timeliness of loads, number of processed records, validation errors
Native monitoring Each workspace has built-in monitoring of item execution, resource use and errors. Use it as a starting point and add your own alerts on workload-specific thresholds.

See also Operate your workload.

Pipeline observability with Grafana (optional)

When the observability tier is deployed, you get an Azure Managed Grafana instance with a dashboard and an alert on failed Fabric pipeline runs. How it is deployed is on Observability.

flowchart LR
    p["Fabric pipelines"] --> lh["pipeline_logs Lakehouse"]
    lh -->|"log ingestion pipeline (your team)"| law["Log Analytics: FabricPipelineRuns"]
    law --> g["Managed Grafana: dashboards"]
    law --> a["Alert rule: pipeline failures"]
    a --> ag["Action group: email"]

The dashboards stay empty until your team feeds them

The dashboards read the FabricPipelineRuns table in Log Analytics. The Grafana deployment does not create or fill it. Your data engineering team builds the log ingestion pipeline. Until it runs, the panels show "No data".

Access

Sign in with your Entra ID account (MFA applies). There are no local Grafana accounts and no API keys by default. The URL is given by the platform team and has the form https://grafana-<workload>-<env>-<region>.<id>.grafana.azure.com. In private mode you need VPN or ExpressRoute.

Access is by three dedicated Entra ID groups, separate from the Fabric workspace groups:

Role For Can
Viewer Business stakeholders, management View dashboards and alerts, use time range, filters and drill-down
Editor Data engineers and analysts Everything a viewer can, plus create and edit dashboards and manage alert rules
Admin The platform team Everything an editor can, plus data sources, plugins, notification channels, users and organisation settings

A user holds one Grafana role. Request access through Request and change: name the person and the role. The platform team adds the person to the group, and access applies after the group membership propagates (the source mentions 5 to 10 minutes). To remove access, request removal from the group.

The default dashboard

Fabric Pipeline Observability has these panels:

Panel Shows
Pipeline failures (last 24 h) Count of failed runs. Green at 0, yellow at 1 to 4, red at 5 or more.
Pipeline runs by status Share of succeeded, failed, in-progress and cancelled runs
Pipeline execution timeline Runs per 15 minutes, split by status
Recent failed pipelines The last 50 failed runs with time, pipeline, run ID, duration and error message
Average pipeline duration (7 days) Hourly average per pipeline, to spot slower pipelines
Pipeline success rate (7 days) Gauge. Green at 95% or more, yellow at 80% or more, red below 80%.

Dashboards refresh every 30 seconds. Dashboards that are deployed from code cannot be edited in the Grafana UI: change the code. Dashboards that an editor creates in the UI are not in code and do not survive a redeployment.

Your own dashboards

In the UI: add a visualisation, choose the Azure Monitor data source, query type Logs, select the Log Analytics workspace and write the KQL. Save it in a folder.

As code (preferred for dashboards that must stay): export the dashboard JSON, replace the Log Analytics resource ID with the template variable ${LAW_RESOURCE_ID}, escape Grafana template variables such as ${DS_AZURE_MONITOR} as $${DS_AZURE_MONITOR}, and place the file in a sub-folder of the dashboards directory of your configuration repository. Each sub-folder becomes a Grafana folder and each .json file a dashboard. Run the pipeline.

Example queries:

// Runs per day by status
FabricPipelineRuns
| where TimeGenerated >= ago(30d)
| summarize Count = count() by bin(TimeGenerated, 1d), Status
| order by TimeGenerated asc

// Ten slowest successful pipelines
FabricPipelineRuns
| where TimeGenerated >= ago(7d) and Status == "Succeeded"
| extend DurationSeconds = datetime_diff('second', EndTime, StartTime)
| summarize AvgDuration = avg(DurationSeconds), MaxDuration = max(DurationSeconds) by PipelineName
| top 10 by AvgDuration desc

Alerts

The default alert fires when a pipeline run has failed.

Setting Default Configurable
Evaluation interval every 5 minutes yes
Lookback window 1 hour yes
Condition any failed pipeline run no
Severity 1 (Error) no
Auto-mitigation the alert resolves when the next window has no failures no
Notification email through an Azure Monitor action group yes (addresses)

Without email addresses the alert still fires and is visible in Azure Monitor, but nobody receives an email. Receivers other than email (SMS, webhook, Logic App) are not set by the configuration: the platform team adds them to the action group in the portal. Editors can create more alert rules in the Grafana UI.

Limits to know

  • One role per user, and no folder-level permissions: every user with a role sees all dashboards, and editors can edit all of them. Folders only organise. If you need separation, ask for a separate Grafana instance for that team.
  • The default dashboard uses the first Log Analytics workspace of the list. Dashboards over several workspaces need the dashboard JSON edited.
  • After the first deployment, the managed identity roles take up to 10 minutes to propagate. Queries can fail with "Insufficient permissions" in that period.