Data Architecture (Reference Pattern)¶

This page describes a reference data architecture for building a medallion-style data platform on top of the Base Fabric Data Platform. It comes from the architecture diagrams of the solution. The platform delivers the workspaces, Lakehouses, Warehouses, connections and network. It does not deliver or enforce the layers, schemas, pipelines or dbt models below. They are a starting point for the workload team.

Reference design, your design may differ

Treat the names (staging, active, landing, bronze, silver, gold) as a convention to adopt or adapt. The platform does not require them.

Layers¶

flowchart LR
    subgraph src["Sources"]
        s1["Files and blob storage"]
        s2["Databases: Oracle, SQL Server, Azure SQL"]
        s3["API, SharePoint, notebooks"]
    end
    subgraph lh["Lakehouse"]
        stg["Files/staging: full history"]
        act["Files/active: latest only"]
    end
    subgraph wh["Warehouse layers (dbt)"]
        land["Landing"]
        br["Bronze"]
        sil["Silver"]
        gold["Gold"]
    end
    s1 -->|"copy files, shortcut"| stg
    stg -->|"select latest"| act
    s2 -->|"copy job, mirroring"| land
    s3 -->|"notebook ingestion"| stg
    act -->|"external tables"| land
    land --> br --> sil --> gold
    gold --> serve["Serve: Power BI, ML, API"]

Layer	What it holds	Notes
Lakehouse `Files/staging`	The full historical archive of file-based sources	Path `<source>/<table>/<date>/*.parquet`. Permanent archive, no separate archive folder. Parquet is columnar, compressed and carries its schema. Used for compliance and replay. An integration storage account is synced here daily (incrementally).
Lakehouse `Files/active`	Only the latest date folder per table	Path `<source>/<table>/.parquet`. Copied from staging with select latest*. Replaced daily. It is the target of the external tables of the landing layer, and replay means copying a specific date from staging.
Landing (Warehouse)	An exact 1:1 copy of the source, for validation	File-based sources: external tables over `Files/active`. Mirrored databases: the SQL analytics endpoint (cross-database query). Copy jobs: native Warehouse tables `landing.<source>_<table>`. No duplication of data for files and mirrored databases. dbt queries all of them in the same way. Tests: schema validation, row counts, data arrival.
Bronze	Native tables built by dbt from all sources	Schema `bronze.<source>_<table>`. One table per source system (source grain). Basic cleaning, type conversion, de-duplication. Tests: uniqueness, not null, valid values.
Silver	Business entities	Schema `silver.<entity>`, for example an assets table or a maintenance history table. Joins, lookups, business rules and calculated fields, integrated from all sources. The single source of truth for business data, ready for consumption or for aggregation into gold. Tests: referential integrity and business rules.
Gold	Aggregated, consumption-ready data	The diagram shows a gold Warehouse layer that feeds serving.

Ingestion methods¶

Method	Use it for	Lands in
Copy files	File sources: load into Lakehouse files, then load to table	`Files/staging`, then `landing`
Notebook ingestion	Sources that need code, such as APIs	Staging files or `bronze.<source>_<table>`
Shortcut	Data that already lives elsewhere and must not be copied	`Files/<shortcut>/`, then external tables
Copy job	Relational sources, copied as a whole or incrementally	`landing.<source>_<table>` as native tables
Database mirroring	Databases that Fabric can mirror (the diagram shows Azure SQL Database)	Mirrored database, queried through its SQL endpoint, as `<mirrored_db>.<table>`

The diagram lists example sources: Oracle, SQL Server (on-premises), blob storage, SharePoint, an API, notebooks and Azure Databricks. On-premises and private Azure sources are reached through the VNet data gateway or managed private endpoints (Connectivity).

Environments and promotion¶

The second diagram of the solution shows how development, pre-production and production relate.

flowchart LR
    dev["Dev workspace (linked to Git, dev branch)"] -->|"pull request and merge"| pre["Pre-prod workspace (preprod branch)"]
    pre -->|"pull request and merge"| prod["Prod workspace (prod branch)"]
    tf["Terraform and CI/CD: platform team"] -.->|"deploys infrastructure"| dev
    tf -.-> pre
    tf -.-> prod

The platform team deploys the infrastructure (capacities, workspaces, connections) with Terraform and CI/CD, first the core components and then the workspaces and capacities.
Data engineers develop in the development workspace, connected to a Git branch. Data models are promoted by pull request and merge to the pre-production and then the production branch.
Production workspaces are not linked to Git directly in the basic setup. They are updated by the pipeline.

Platform building blocks the pattern relies on¶

Need	Provided by
Workspaces with OneLake, Lakehouses and Warehouses	Service description
Private access to sources	Connectivity
Git version control	Connectivity
Pipeline health monitoring	Monitoring
Catalog, classification and lineage	Governance

dbt jobs, the integration storage account and the transformation schedules are not deployed by the platform. Agree with the platform team who sets them up.