Built for Agents: The Case for Spark Declarative Pipelines

Databricks reports that AI agents now create over 70% of Spark Declarative Pipelines, so cheap iteration and readable code matter more. A benchmark of procedural PySpark against declarative pipelines found the declarative style cheaper to validate, shorter, and free of in-memory shortcuts, while logic errors stayed similar.

• Each of 22 tasks used 12 datasets, yielding 264 builds per approach with Opus 4.8.
• Declarative dry-run validation caught structural errors, and 55 of 264 stopped before any output.
• 19 procedural builds processed rows in memory, and no declarative build used that shortcut.

• over 70% — Share of SDP pipelines on Databricks now created by agents.
• 596 executor-seconds — Procedural cluster time across the builds that were measured.
• 17 — Declarative run executor-seconds across the same measured builds.
• 34X — Reported compute reduction from declarative dry-run validation versus procedural runs.
• 64 lines — Median size of completed declarative pipelines in the benchmark.
• 131 — Median lines for completed procedural pipelines in the benchmark.

Benchmark covered 22 tasks and 12 datasets per task with Opus 4.8.

Dry runs catch structural mistakes only. Silent-error rates stayed within roughly 15% of each other, too small to count as a real difference.