Information security for software delivery

ISO/IEC 27001
ISO/IEC 27001 certified
The order of decisions
Laying out a Databricks lakehouse before any pipeline exists
Each stage constrains the next, which is why they are decided in this order. Retrofitting the catalogue under a year of ungoverned notebooks is the expensive version of the same work.
01
Workspaces by environment
Development, staging and production separated at the workspace level, so an experiment can never touch a serving table by accident.
- Workspaces
- Repos
02
Catalogue by domain
Unity Catalog hierarchy mapped to business domains with access granted at the schema level — ownership expressed in the platform, not in a wiki.
- Unity Catalog
- Groups
03
Refinement in layers
Raw data lands untouched, is cleaned and conformed in a middle layer, and is served from tables shaped for consumption — so every number is traceable back through the layers.
- Delta tables
- Pipelines
04
Serving, per consumer
BI tools read SQL warehouses; data scientists read feature tables; downstream systems read published schemas — each consumer gets a surface shaped for it, from the same governed core.
- SQL warehouse
- Serving
The cost guardrails
Compute policies run alongside everything: cluster sizes capped by role, auto-termination enforced, jobs on job compute rather than always-on clusters, and spend tagged by team so the bill has an owner per line.
None of this slows the first pipeline down by more than days; all of it is painful to add under a year of accumulated notebooks.
The fit
When one platform for engineering and machine learning earns its keep
The honest case for Databricks is workload mix. An organisation whose data work is purely SQL analytics on structured data is often better served by a plain warehouse — we say so on our Snowflake page, and we mean it in both directions. Databricks earns its keep when the mix is genuinely varied: heavy transformation pipelines in Python or Scala, machine learning that needs the same data the analysts query, streaming alongside batch, large volumes of semi-structured or file-based data that would be awkward and expensive to force through a warehouse’s front door. One platform carrying that mix beats three platforms and the pipelines between them.
The second honest question is the team. Databricks assumes engineers — people comfortable with notebooks, Spark, version control and infrastructure-as-code. A team of SQL analysts can work in it through the SQL warehouse surface, but if that surface is all you will use, you are paying lakehouse complexity for warehouse behaviour. Part of our scoping is saying which platform your actual team, doing your actual work, will be productive in — and we would rather lose a Databricks engagement than build one the client’s team cannot operate.
The bill
What actually moves a Databricks bill, and the discipline that holds it
Databricks charges for compute time, so the bill is a mirror of engineering habits. These are the drivers we put policies around from the first week.
| Variable | Effect on price | Why |
|---|---|---|
| Clusters left running | The classic silent cost | An all-purpose cluster someone forgot on a Thursday bills through the weekend. Auto-termination enforced by policy, not by memory, removes the whole category. |
| Jobs on interactive compute | Paying the premium rate for scheduled work | Scheduled pipelines belong on job compute, which costs meaningfully less than the interactive clusters meant for humans. Moving them is usually the fastest saving in an unmanaged account. |
| Oversized by default | Every job inherits the big template | When the shared cluster template is sized for the largest job, every small job pays for it. Right-sizing against measured runs, with policies capping what a role may launch, keeps size proportional to work. |
| Data laid out badly | Every query scans more than it should | Unpartitioned tables, small-file sprawl and missing clustering make every downstream job pay a scanning tax forever. Table layout is a cost decision, not just a performance one. |
Ask any supplier quoting this
- Which of our workloads would run on job compute, and what share of current spend is interactive clusters?
- What compute policies will exist on day one, and who can launch a cluster outside them?
Our role
What we build on Databricks and what your platform team takes over
- 01
The layout before the code
Workspaces, catalogue hierarchy, access model and compute policies designed and reviewed with you before the first pipeline — the decisions that are cheap now and expensive later.
- 02
The pipelines, engineered not scripted
Version-controlled, tested pipelines through the refinement layers, with data quality checks between them — built as software, because that is what they are.
- 03
The bill, observable from week one
Spend tagged by team and job, dashboards on cost per pipeline, and the policies that keep the account from drifting — so finance never meets a surprise it cannot trace.
We build the lakehouse so your platform team can run and extend it without us — the layout documented, the policies enforced in code, and the first pipelines as worked examples of the standard.
Common questions
What buyers ask
Do we need Databricks if we already run a data warehouse that works?+
Only if your workloads have outgrown what a warehouse does well. A warehouse that serves your SQL analytics reliably is not a problem to solve. The signals that a lakehouse has become worth its complexity are concrete: machine learning teams exporting data out of the warehouse to work on it, transformation logic that fights SQL and wants a real programming language, streaming requirements bolted awkwardly onto batch, or large semi-structured data that is expensive to load and store in warehouse formats. If none of those is true, keep the warehouse — and if some are, the answer may still be adding Databricks beside it rather than replacing it.
How do we stop Databricks costs drifting up month after month?+
By making cost a property of the platform’s configuration rather than of engineers’ vigilance. Compute policies cap what each role can launch; auto-termination is enforced, not suggested; scheduled work runs on job compute; and every job and cluster carries tags that land in a cost dashboard someone reviews. Drift almost always comes from the gap between what is allowed and what is intended — a bigger cluster than the job needs, an interactive cluster doing scheduled work, a forgotten notebook. Close the gap with policy and the drift stops being a monthly discovery.
Can our SQL analysts work on Databricks, or is it for engineers only?+
They can — the SQL warehouse surface gives analysts a familiar SQL editor, BI tools connect to it the way they connect to any warehouse, and Unity Catalog governs what each analyst sees. The distinction that matters is who does what: analysts consume the governed serving layer through SQL, while building and changing the pipelines that produce that layer remains engineering work. Where it goes wrong is when analysts are expected to maintain notebook pipelines as a side duty, or when the platform is bought for a team that will only ever use the SQL surface — the first is unfair to the analysts, the second is paying for capability nobody uses.
In this cluster
Plan your lakehouse layout with us
A lakehouse design session that produces your workspace and catalogue layout, the first pipeline’s shape, and the compute policies before any cluster spins up.
Start the conversation