Fabric unifies storage, compute and BI. WhereScape turns that promise into a repeatable delivery system: model, generate, deploy, document; on rails. This blog gives you a field-tested blueprint you can adopt tomorrow.
The target architecture (at a glance)
- Storage: OneLake (Delta).
- Compute: Fabric Warehouse (lake-centric SQL), Spark as needed.
- Integration: Fabric Data Factory (pipelines/dataflows).
- Governance: Microsoft Purview (catalog, lineage, classification).
- Consumption: Power BI (semantic models, Direct Lake).
- Automation: WhereScape for metadata-driven design, code gen, orchestration, docs.
- Lifecycle: Git-backed assets, CI/CD across Dev → Test → Prod.
Step-by-step blueprint
1) Establish environments & access
- Workspaces: Dev/Test/Prod in Fabric; map to OneLake folders; define data domains (Sales, Finance, Ops).
- Security: Entra ID groups for RBAC; enforce least privilege across Fabric and Purview.
- Connectivity: Register sources (SQL Server, SaaS apps, files); store secrets in Key Vault and reference from WhereScape.
2) Harvest and align metadata
- Scan with Purview, import to WhereScape 3D: schemas, column types, sensitivity and glossary terms.
- Normalize naming standards: business-friendly names for datasets and tables; document them once in metadata.
- Decide architecture per domain: DV2 vs direct star; don’t over-engineer low-change domains.
3) Ingest (Bronze) fast—at scale
- Bulk-generate pipelines in Fabric Data Factory from WhereScape metadata (50 tables in minutes, not weeks).
- Set load patterns: full vs incremental/CDC; file landing conventions; schema drift handling.
- Observability: standardize metrics (rows read/written, latency, failures) and alerts from day one.
4) Standardize & conform (Silver)
- Apply cleansing and type harmonization with generated ELT.
- Conform dimensions (Date, Product, Customer) early to reduce downstream reconciliation.
- Automate tests: row counts, not-null, referential checks; WhereScape surfaces failures with clear lineage back to source.
5) Curate for consumption (Gold)
Two main patterns:
- Data Vault 2.0 core → generated stars for BI.
- Direct stars where DV2 is overkill.
Either way, WhereScape generates Warehouse DDL + load logic and keeps documentation/lineage synchronized.
6) Semantic models & Power BI
- Model alignment: publish clean star schemas; generate or template semantic models (datasets) via metadata.
- Connectivity modes: enable Direct Lake for hot, high-volume facts; mix with Import where needed (complex calc/row-level security).
- Governance for BI: certify datasets; propagate glossary terms and sensitivity labels from Purview.
7) Orchestration & scheduling
- Dependency graph: WhereScape orders tasks Bronze→ Silver→ Gold→ BI refresh; retries with exponential backoff.
- Windows & SLAs: reflect business timings (e.g., sales close) and back up with alert policies (Ops, Slack/Teams).
- Backfills: parameterized historical loads without one-off scripting.
8) DevOps & CI/CD
- Version everything: WhereScape model files, SQL, pipeline JSON, Power BI artifacts.
- Branching model: feature branches → PR → automated checks (linters/tests) → promote to Test/Prod.
- Release automation: YAML/GitHub Actions/Azure DevOps to deploy generated assets; keep Purview lineage in sync as part of the pipeline.
9) Security & compliance by construction
- Column/classification alignment: sensitivity labels from Purview flow into masking rules/row-level security templates.
- Data sharing: OneLake shortcuts for cross-domain sharing without copies; WhereScape documents usage and lineage.
- Audit pack: generate PDF/HTML documentation each release (objects, lineage graphs, owners, SLAs, tests).
10) Cost & performance hygiene
- Partition & clustering guidance baked into templates based on size/usage.
- Prune & cache: favor predicate pushdown; materialize heavy aggregations where it pays back.
- Capacity planning: monitor Warehouse and Power BI capacity; right-size, don’t over-provision.
- Data minimization: only land/retain what you need; automate retention policies per domain.
A 30 – 60 – 90 day plan: realistic, not heroic
Days 1–30: Foundation & first value
- Environments, security, Purview scan connected.
- 2–3 sources ingested to Bronze; Silver standardization in place.
- First Gold star (or DV2 + star) published; a Power BI report live on Direct Lake.
- Lineage visible end-to-end; basic alerting operational.
Days 31–60: Scale patterns
- Add 4–6 new sources using the same templates.
Introduce DV2 where agility/history required; generate 2–3 marts. - CI/CD wired; weekly releases; audit pack and data quality dashboards in place.
Days 61–90: Industrialize
- Domain ownership (Data Mesh) formalized; shared conformed dimensions.
- Self-service BI on certified datasets; governance council reviews lineage & quality KPIs monthly.
- Cost controls (partitioning, caching) tuned; SLA adherence > 95%.
KPIs that prove it’s working
- Lead time to first report: < 6 weeks from project kickoff.
- Manual code eliminated: ~95% of boilerplate (pipelines/DDL/ELT) generated.
- Data quality: < 0.5% failed checks per run; MTTR < 2 hours with clear root-cause lineage.
- Adoption: # of certified datasets, # of active BI users, refresh success rate.
- Cost: $/query trend stable or improving as volumes grow.
What this looks like day-to-day (roles)
- Data architects: design in WhereScape 3D, choose patterns (DV2 vs star), set standards once.
- Engineers: connect sources, tune templates, watch dashboards—not hand-write plumbing.
- BI developers: focus on business logic and UX; semantic models aren’t wrestled into shape.
- Stewards/compliance: live in Purview; lineage and glossary stay current automatically.
- Ops: manage a small, predictable set of runbooks; most failures self-diagnose via generated metadata.
Mini FAQ
- Do we need Spark? Often not for the first releases. Warehouse + generated ELT covers most needs; add Spark selectively.
- Can we mix patterns? Yes. Use DV2 for volatile domains; simple stars elsewhere. Templates support both in one program.
- Will governance slow us down? Not if it’s automated. WhereScape publishes lineage and docs as part of deployment, not after.
Final takeaway
Fabric provides the platform. WhereScape provides the operating system for delivery—turning source metadata into governed datasets and reports with speed, consistency and auditability. Follow this blueprint and you’ll move from pilot to production without the detours that sink so many platform programs.
Want this blueprint tailored to your domains? Book a 20-minute working session—we’ll map your 30-60-90 day plans and show the automation live on Microsoft Fabric.
About the Author
Patrick O’Halloran is a Senior Solutions Architect at WhereScape with over two decades of experience in data warehousing and analytics. He works with global organizations to implement automated data infrastructure using WhereScape RED and 3D, helping teams scale their data operations efficiently and reliably.
