Apache Spark 4.2: what's new for analysts and data engineers

Apache Spark 4.2.0 was released on 14 July 2026, the third release of the 4.x line, with more than 1,700 tickets from over 250 contributors. It is now arriving on the platforms most teams actually use: Databricks shipped it in Databricks Runtime 19.0, while Microsoft Fabric's new Runtime 2.0 is on Spark 4.1. If you write Spark SQL or PySpark, this is the release that makes everyday work noticeably nicer, and it brings change data capture and a semantic layer into open-source Spark.

QUALIFY, at last

If you have used Snowflake, BigQuery or Databricks SQL, you already know QUALIFY: it filters on the result of a window function, the way HAVING filters on an aggregate. Open-source Spark SQL now supports it, so the classic "latest row per customer" query loses its subquery.

SELECT customer_id, order_id, order_date
FROM orders
QUALIFY ROW_NUMBER() OVER (
  PARTITION BY customer_id ORDER BY order_date DESC
) = 1;

SELECT customer_id, order_id, order_date
FROM (
  SELECT *,
         ROW_NUMBER() OVER (
           PARTITION BY customer_id ORDER BY order_date DESC
         ) AS rn
  FROM orders
) t
WHERE rn = 1;

Other SQL quality-of-life changes: SQL scripting gained cursors, pipe syntax accepts | as well as |> and now supports aggregates, PIVOT accepts aliases, DROP TABLE can drop a view, there is a time_bucket function for grouping timestamps into fixed buckets, and array_agg, collect_list and collect_set support IGNORE NULLS. For a refresher on window functions, see SQL window functions explained.

The headline features

Change data capture becomes a first-class feature

Most pipelines spend a surprising amount of code working out what changed since yesterday. Spark 4.2 standardises it. Connectors can now expose a change feed, Spark SQL reads it with a new CHANGES clause, PySpark gets a matching changes() API, and streaming reads can return net changes. On top of that, Auto CDC in Spark Declarative Pipelines applies those changes to a target table as a slowly changing dimension Type 1, so the latest version of each row wins without writing MERGE statements by hand.

Metric views: a semantic layer inside Spark

Metric views let you declare measures and dimensions once, with CREATE VIEW ... WITH METRICS, and reuse them across SQL, dashboards and AI agents. It is the same idea as the dbt Semantic Layer, Snowflake's semantic views and Power BI semantic models: one definition of "revenue" instead of five. If you are new to this way of modelling, our guide to star schemas and dimensional modelling explains the facts-and-dimensions thinking behind it.

PySpark: faster by default

Arrow-optimised Python UDFs and Arrow-based data exchange between the JVM and Python are now on by default. Arrow moves data in columns instead of row by row, which is usually much faster. PySpark also gained DataFrame.zipWithIndex, vector similarity and distance functions, and Real-Time Mode for stateless queries.

pip install --upgrade "pyspark==4.2.0"

Streaming and performance

• Stable source names: name a streaming source and you can add, remove or reorder sources without breaking checkpoints.
• Safer state: automatic snapshot repair and row checksums detect corrupted state.
• Faster plans: join pushdown through UNION, whole-stage code generation for UNION, faster window frames and Parquet reading, and ORC serialisation of nested types roughly halved in time.
• Spark Connect: head(), take() and tail() no longer scan the whole table.

Where you can use it

Working in Python more than Spark? See what's new in Polars 2.0 and the pandas 3.0 upgrade guide. For the lakehouse table format underneath, read Iceberg v3 explained.

Sources: the Apache Spark 4.2.0 release notes and the Databricks blog post "Introducing Apache Spark 4.2" (16 July 2026). Check the full release notes for every behaviour change before upgrading.

Alessandro Ecclesie AgazziWritten by Alessandro Ecclesie Agazzi, analytics engineer in London. Updated 17 October 2026. About me