What is Lakeflow SDP?

Lakeflow Spark Declarative Pipelines (SDP) is the rename of Delta Live Tables. You declare the tables and the transformations between them; the engine handles ordering, incremental computation, retries, and quality enforcement. It is the default way to build bronze → silver → gold on Databricks in 2026.

How does SDP differ from regular Spark Structured Streaming?

You write the transformation, not the orchestration. SDP figures out which tables are streaming vs materialized, how to checkpoint, when to refresh, and how to apply quality expectations. You stop maintaining the glue code and start declaring the lineage.

What are expectations in SDP?

Inline data-quality constraints, `EXPECT amount > 0` or `EXPECT_OR_DROP customer_id IS NOT NULL`. SDP enforces them during ingestion: track-only, drop-bad-rows, or fail-the-update. The result is a quality contract that lives next to the transformation instead of in a separate dbt test or a downstream alert.

When should I use SDP vs dbt on Databricks?

dbt is great when your team is already deep in SQL-first development and you want batch warehouse-style modeling on a SQL Warehouse. SDP wins when you have streaming or near-real-time data, want incremental materialization out of the box, or want a single declarative description of bronze → silver → gold. They coexist, dbt for the gold layer marts, SDP for the streaming bronze/silver pipeline that feeds it, is a common combination.

Can SDP run on serverless compute?

Yes, and that's the default in 2026. Serverless SDP pipelines start faster, scale better under bursty load, and remove the cluster-tuning step entirely. Classic compute remains available for workloads that need specific runtime versions or VPC/customer-managed-key configurations.

How do I do change data capture in SDP?

Use `APPLY CHANGES INTO` for merge-style CDC from CDC sources (Debezium, Lakeflow Connect ingestion). It handles late-arriving events, type-2 history if you ask for it, and dedup. The whole CDC story collapses into ten lines of SQL.

Topic guide

· Covered in Volume 3 · The Production Lakehouse

Lakeflow Spark Declarative Pipelines (SDP) on Databricks (2026)

Lakeflow Spark Declarative Pipelines, the platform formerly known as Delta Live Tables, is how data engineering on Databricks gets written in 2026. You declare the tables and the transformations; the engine handles incremental refresh, streaming, quality enforcement, change data capture, and the dependency graph. This is the practitioner-level guide: when to reach for SDP, what its operational shape is, where it beats dbt and where they pair, and the patterns that hold up at scale.

9 min read Updated 2026-05-18 By Ritesh Modi

The shape of data engineering on Databricks shifted twice in three years. First when Delta Live Tables became production-grade. Second when it was rebranded Lakeflow SDP and folded into the broader Lakeflow family alongside Lakeflow Connect (ingestion) and Lakeflow Jobs (orchestration). The model didn't change; the positioning did. SDP is no longer "the new way to do batch ETL"; it's the default declarative compute model for any pipeline that goes bronze → silver → gold.

What you actually write

A pipeline is a notebook (or a set of them) where each cell declares a table, streaming or materialized, and the transformation that produces it. The engine reads the whole notebook, builds the DAG, figures out which tables need full refresh and which can be incremental, schedules the work, runs the quality expectations, and writes everything into Unity Catalog. You write the SQL or PySpark. The runtime does the orchestration.

Two refresh shapes: streaming tables append per row, materialized views recompute on schedule — Figure 32.1 · Volume 3, Chapter 32 One input, two refresh shapes. Streaming tables append every arriving row; materialized views recompute the whole result on schedule. Cost scales differently, pick the shape that matches the workload.

Streaming tables vs materialized views

Two table types matter. A streaming table is append-only and incremental, it processes new rows since the last update. A materialized view recomputes its definition either incrementally (where SDP can prove correctness) or as a full refresh. Most bronze layers are streaming tables (raw events landing forever). Most silver layers are streaming tables with expectations and dedup. Most gold layers are materialized views (joined aggregations that get queried by dashboards).

Expectations: the quality contract that lives with the code

Inline expectations are the single biggest reason teams adopt SDP after they've tried building the same thing in raw Spark. Three modes: track-only (record violations, don't drop), drop-bad-rows (drop the row, log it), and fail-the-update (halt the pipeline). The right mix is per-pipeline: bronze gets track-only because the goal is fidelity; silver gets drop-or-fail because the goal is quality; gold rarely needs them because the silver contract already enforced what mattered. The expectations show up in the SDP UI and in lineage automatically, your data quality is no longer a separate dashboard you forget to look at.

APPLY CHANGES INTO: the CDC story

Change data capture used to be a chapter-long discussion of merge windows, type-2 dimensions, and late-arriving event handling. APPLY CHANGES INTO collapses most of that into a single declaration:

APPLY CHANGES INTO target_table
FROM stream(cdc_source)
KEYS (customer_id)
APPLY AS DELETE WHEN op = 'D'
SEQUENCE BY event_ts
COLUMNS * EXCEPT (op)
STORED AS SCD TYPE 2;

SDP handles deduplication, ordering by sequence column, late-arrival re-application, and the SCD2 history. The result is that CDC stops being a project and becomes a pattern.

SDP and Lakeflow Connect: ingestion that doesn't fight back

Lakeflow Connect lands managed connectors for Salesforce, Workday, ServiceNow, MySQL, Postgres, SQL Server, and a growing list of SaaS sources directly into UC tables. The end-to-end pattern in 2026 is: Lakeflow Connect → bronze streaming table → SDP transforms to silver and gold → Lakeflow Jobs orchestrates and triggers downstream. No more bespoke ingestion code. No more cron-driven sync jobs. The whole ingest layer becomes declarative.

Where dbt fits

Teams already heavily invested in dbt don't need to abandon it. dbt-databricks works against SQL Warehouses and remains the cleanest tool for modeling-style transformations done by analytics engineers in SQL. The pattern that works: SDP owns the streaming and quality-critical bronze/silver layer; dbt owns the gold-layer marts that analysts iterate on. Both write into UC; lineage stitches automatically.

Operational shape

SDP pipelines deploy as Databricks Asset Bundle artifacts. They run on serverless compute by default. Triggered (event-driven) and continuous modes are both available; continuous is for sub-minute latency, triggered is for the 90% of pipelines that just need to fire on data arrival or a schedule. The Lakeflow Pipeline UI surfaces lineage, run history, expectation violations, and per-table refresh status, the operational view you used to assemble from five separate dashboards.

What's in Volume 3 on this

Two chapters and the running retail-analytics example. Chapter 31 covers ingestion with Auto Loader and Lakeflow Connect, the source layer that feeds SDP. Chapter 32 is SDP end-to-end: streaming tables, materialized views, expectations, APPLY CHANGES, and a full worked pipeline that takes the retail event stream from raw landing to gold-layer aggregations ready for Lakeflow Jobs to schedule downstream consumers.