What's new in Apache Spark 4.3: real-time state, SCD Type 2 and ASOF JOIN

Apache Spark 4.3.0 came out on 30 October 2026, only three and a half months after Spark 4.2. That is not an accident: from this release on, Spark ships a feature release every quarter instead of every six months. On 5 November Databricks announced that it is available in Databricks Runtime 19. The release closes more than 1,100 Jira tickets from over 210 contributors. Below are the changes that matter for analysts and data engineers, plus what we saw when we ran PySpark 4.3 on a small 2-core machine.

Real-Time Mode now remembers

Real-Time Mode arrived in earlier releases for simple, stateless streams. In 4.3 it handles the jobs that need memory: a running count per device, removing duplicate events, or a fraud check that looks at the last few transactions. Spark achieves this with a new execution model: producer and consumer stages run at the same time, connected by a "pipelined shuffle" that passes records along as they arrive instead of waiting for a whole stage to finish. The code you write is still ordinary Structured Streaming; you only change the trigger.

counts = events.groupBy("device_id").count()

query = (
    counts.selectExpr("CAST(device_id AS STRING) AS key",
                      "CAST(count AS STRING) AS value")
    .writeStream.format("kafka")
    .option("kafka.bootstrap.servers", "localhost:9092")
    .option("topic", "device-counts")
    .option("checkpointLocation", "/checkpoints/device-counts")
    .outputMode("update")
    .trigger(realTime="5 minutes")   # checkpoint interval, not latency
    .start()
)

SCD Type 2 without the MERGE headache

A "current state" table tells you where a customer lives today, not where they lived when they placed an order. Keeping history used to mean writing your own MERGE logic. Spark 4.3 adds SCD Type 2 to Auto CDC in Spark Declarative Pipelines: every change creates a new version of the row, with __START_AT and __END_AT columns marking when it was valid.

CREATE STREAMING TABLE customer_history;

CREATE FLOW customer_history_cdc AS AUTO CDC INTO customer_history
FROM STREAM(customer_changes)
KEYS (customer_id)
SEQUENCE BY event_time
STORED AS SCD TYPE 2
TRACK HISTORY ON (status, region);

The same is available in Python with stored_as_scd_type=2. One condition: the target table must live in a catalog that supports row-level operations, and plain Parquet tables do not. In practice that means Apache Iceberg in open-source Spark, or Delta tables on Databricks.

Project Feather: Spark for small data

Spark has always paid a fixed cost for every query: planning, scheduling tasks and shuffling data between them. On a 1,000-row sample that overhead is most of the time. Project Feather starts to remove it. With one setting, small queries that read a single small file can run as one task, with no shuffle.

spark.conf.set("spark.sql.optimizer.singleTaskExecution.enabled", "true")
spark.read.parquet("small.parquet").groupBy("category").count().show()

We tried it on a 900-row Parquet file. The physical plan lost its Exchange step and the fastest of eight runs dropped from 85 ms to 53 ms. For comparison, DuckDB 2.0 and Polars 2.1 answered the same query in about 1 ms, and starting the local Spark session took 5.3 seconds. Feather makes Spark more pleasant for testing code on samples, especially code suggested by an AI assistant, but it does not turn Spark into a laptop engine. Our DuckDB vs Polars vs Spark comparison has the full numbers.

SQL improvements

SELECT title, content
FROM JSON_TABLE(
  '{"sections":[{"title":"Returns","text":"Return within 30 days."},
                {"title":"Delivery","text":"Ships in 2 days."}]}',
  '$.sections[*]' COLUMNS (
    title   STRING PATH '$.title',
    content STRING PATH '$.text'
  )
) AS s;
-- Returns  | Return within 30 days.
-- Delivery | Ships in 2 days.

SET spark.sql.join.asofJoin.enabled = true;

SELECT t.trade_id, q.price
FROM trades AS t
LEFT ASOF JOIN quotes AS q
  MATCH_CONDITION (t.ts >= q.ts)
  ON t.sym = q.sym;

Also new: ANSI UNNEST in the FROM clause, time travel with an @ after the table name, COMMENT ON COLUMN, a TIME type down to nanoseconds, and nanosecond-precision timestamps (TIMESTAMP_NTZ(9)) as an opt-in preview.

Smaller changes worth knowing

• Read files inside archives: CSV, JSON, Parquet and more can now be read straight from tar, zip and 7z files (opt-in).
• DataFrame.zip combines two column selections of the same DataFrame without falling back to RDDs.
• Spark Connect gets REST endpoints to list sessions and operations, operation IDs for tracing, and a better JDBC driver with ARRAY, MAP and STRUCT support.
• Faster Parquet reads and several optimizer improvements, such as code generation for UNION by default.
• Experimental: simple Python UDFs, such as len(), can be translated into native Spark expressions.

Before you upgrade

Keep reading: what was new in Spark 4.2, the Spark SQL and PySpark cheat sheet and the Databricks October 2026 update.

Sources: Apache Spark 4.3.0 release notes and news (30 October 2026); Databricks blog, "Introducing Apache Spark 4.3, now available in Databricks Runtime 19" (5 November 2026); our own tests with PySpark 4.3.0 in local mode on a 2-core machine, where timings are indicative only.

Alessandro Ecclesie AgazziWritten by Alessandro Ecclesie Agazzi, analytics engineer in London. Updated 9 November 2026. About me