Azure Databricks is where a huge share of serious Spark work lives, and an enterprise deployment stands on two pillars that have nothing to do with Spark itself: the network architecture, decided at workspace creation and painful to change, and Unity Catalog, the governance layer that decides whether your lakehouse has access control or access folklore.

Network Architecture First

Deploy with VNet injection into your own network, never the managed default VNet, so clusters live in subnets you control, route through your egress design, and reach your private endpoints. Enable secure cluster connectivity (no public IPs on cluster nodes; the control plane relay handles communication outbound), and put private endpoints on the workspace itself for both front end access (users reach the UI over Private Link, no public workspace URL) and back end connectivity. The compute subnets need sizing thought: every cluster node takes an IP, autoscaling multiplies, and undersized subnets fail cluster launches at the worst moment, so give the pair of delegated subnets a /23 or better in busy workspaces. Egress control follows the series pattern, route through the hub firewall with the documented Databricks required destinations allowed, and your data exfiltration story is the network design, not hope.

Unity Catalog: One Governance Plane

Unity Catalog replaces the old per workspace Hive metastore mess with a metastore per region governing all workspaces: catalogs contain schemas contain tables and volumes, identities federate from Entra ID, and permissions are ANSI grants on securables. The structure that scales: catalogs per domain or environment (sales_prod, sales_dev), managed tables in Delta by default, external locations wrapping the ADLS containers you already have, accessed through storage credentials backed by managed identities via the access connector. Grants go to Entra groups, never individuals, and the lineage capture, every table to table flow through Databricks recorded automatically, quietly becomes the most consulted feature during audits and incident forensics.

GRANT USE CATALOG ON CATALOG sales_prod TO `grp-data-engineers`;
GRANT USE SCHEMA ON SCHEMA sales_prod.curated TO `grp-analysts`;
GRANT SELECT ON SCHEMA sales_prod.curated TO `grp-analysts`;
GRANT MODIFY, SELECT ON SCHEMA sales_prod.curated TO `grp-data-engineers`;

CREATE EXTERNAL LOCATION sales_raw
URL 'abfss://raw@stcorpdatalake.dfs.core.windows.net/sales'
WITH (STORAGE CREDENTIAL cred_lake_mi);

Cluster Policies and the Cost Conversation

Unbounded cluster creation rights are how a data platform’s bill doubles in a quarter. Cluster policies are the guardrail: fixed or bounded node types, mandatory autotermination (30 minutes idle, less for interactive), spot instances for non critical workloads, tags enforced for cost attribution, and Photon enabled where it pays for itself, which for SQL heavy workloads it usually does. Steer workloads to the right compute: serverless SQL warehouses for BI and ad hoc SQL, job clusters (created per run, died after) for scheduled pipelines, shared interactive clusters only for genuine exploration. Jobs on all purpose interactive clusters is the classic money leak; a policy that blocks it pays for the time spent writing it within the month. Round it out with system tables for usage analysis, budget alerts per workspace, and the same weekly review ritual as every other platform in this series, because cost governance is a habit, not a feature.

Databricks and yesterday’s Fabric are not either or: plenty of estates run Databricks for heavy engineering and ML with gold tables surfaced to Fabric and Power BI through OneLake shortcuts or Delta sharing, one lake underneath, each engine doing what it does best.

Cheers
Osama