Data platform design & build
A governed lakehouse your team can run, with quality and cost control built in from the first pipeline.
One platform, with the unglamorous parts in place: incremental loading, quality gates, access control and a cost ceiling.
What you get
- A medallion lakehouse with one certified layer everyone reports from
- An incremental, re-runnable ingestion pipeline per source system
- Data quality checks with severity-tiered alerting
- Group-based access, row-level security and separation of safeguarding data
- Automated capacity scheduling
- Reporting embedded where your staff already work, without a licence per seat
- A runbook, user guides and a handover somebody can maintain
- Who it is for
- Organisations whose data sits in several systems that do not talk to each other, whose reporting is assembled by hand, and who have no data team or one person holding it together.
- Shape and duration
- Typically twelve to twenty weeks, phased, with written sign-off at the end of each phase. Discovery first if the shape is not yet clear.
- Tooling I have delivered on
-
- Microsoft Fabric
- Azure
- ADLS Gen2
- Delta Lake
- Databricks
- dbt
- Power BI
- Key Vault
- Entra ID
- Purview
How the platform is built
A medallion lakehouse: raw data landed as it arrived and never altered, a cleansed middle layer that carries history, and a certified layer everyone reports from. Each source system gets its own incremental pipeline that can be re-run safely.
- Watermark-based incremental loads, so a failed run is fixed by re-running it
- Every record landed with its source and load time
- Schema changes detected and flagged
- Full run logging, so you can answer why a number changed
Data quality and observability
Quality checks are configuration, so adding a rule does not mean a deployment. Checks run at every layer boundary with severity tiers: a critical failure stops promotion, a warning flags and lets it through.
- Checks defined once and applied across layers
- Severity-tiered alerting that reaches a person
- Quality control in place before the first report is published
- Pipeline health, run history and cost in one place
Security and cost
Access is granted to groups, so leavers and movers are handled by your existing joiner process. Special-category data can sit in separate tables and workspaces. Analytics capacity is scheduled to pause outside working hours, and reporting is embedded without a paid seat per person.
- Group-based access and row-level security
- Safeguarding data separated at the storage layer
- Secrets in a managed vault
- Automated capacity pause and resume, with the overnight trade-off decided with you
- Reporting embedded into an existing intranet
Proof
Designed
A charity platform build, currently in delivery: around 65% off cloud analytics capacity cost, and no paid reporting licence per member of staff.
Three systems and a spreadsheet holding it together?
That is the usual starting point. A short call will tell us whether a platform is the answer or whether something smaller would do.
Book a call