October 2026 was a busy month for data engines. DuckDB 2.0 arrived on 21 October as a long-term support release, Polars 2.1 followed on 25 October, and Apache Spark 4.3 on 30 October. All three are free and open source, all three read Parquet and Apache Iceberg, and all three now speak SQL. So which should you use? We installed the new versions, ran the same queries on the same 20 million rows, and wrote down what we learned.
Three engines, three designs
DuckDB 2.0
SQL database in a file- Licence
- MIT
- Style
- SQL first, with Python, R, Java and Node clients
- Runs
- in your process, now also as a small server (Quack)
- Excellent SQL: ASOF and NEAREST joins, triggers, MERGE, a fast VARIANT type for JSON
- Reads Parquet, CSV, JSON, Iceberg, Delta and DuckLake, locally or from cloud storage
- Long-term support until at least October 2027
- One machine only; very large shuffles still need care
Best for SQL people, notebooks, dashboards backends and local analytics.
Polars 2.1
DataFrame library- Licence
- MIT
- Style
- Python expressions, plus a growing SQL interface
- Runs
- in your process, with a streaming engine by default
- Fast, strict and predictable: types and errors are explicit
- Group-bys can spill to disk when memory runs out
- Writes and manages Iceberg tables
- Requires Python 3.11 or newer; some new APIs are marked unstable
Best for Python developers replacing pandas.
Spark 4.3
Distributed engine- Licence
- Apache 2.0
- Style
- SQL, PySpark DataFrames, Scala and Java
- Runs
- on clusters, from Databricks, Fabric, EMR, Dataproc or your own
- Scales across many machines, to hundreds of terabytes
- Real-time streaming with state, now around 150 ms P99
- The standard on Databricks and Microsoft Fabric
- Seconds of start-up and per-query overhead on small data
Best when one machine is not enough.
The benchmark
We generated a Parquet file with 20 million sales rows (236 MB, four columns) and a small table of 1,000 stores, then ran two queries in each engine, reading the Parquet files every time. We report the fastest of three runs. The machine was a modest cloud VM with 2 CPU cores and 7 GB of memory, so absolute numbers will be higher or lower on yours; the ranking is what matters.
Fastest of 3 runs. DuckDB 2.0.0, Polars 2.1.0, pandas 3.1.0, PySpark 4.3.0 in local mode with 2 cores. Each run reads the Parquet file.
Same machine and method. The join adds 1,000 store rows to 20 million sales rows.
The pattern is clear. On a single machine, DuckDB and Polars are in a class of their own, and which one is faster depends on the query: Polars edged ahead on the many-groups aggregation, DuckDB was more than twice as fast on the join. pandas, which loads everything into memory first, was 2.5 to 8 times slower. Spark was the slowest here, but that is expected: it is built to spread work across many machines, and on two cores you mostly measure its coordination cost.
Small queries: where overhead shows
Spark 4.3's new Project Feather setting cut the tiny query from 85 to 53 ms by skipping the shuffle, a real improvement for testing code on samples. But DuckDB and Polars answer in about a millisecond, and starting a local Spark session took 5.3 seconds before any query ran. If your data fits on a laptop, that difference adds up all day.
Beyond speed
| DuckDB 2.0 | Polars 2.1 | Spark 4.3 | |
|---|---|---|---|
| Main interface | SQL | Python expressions | SQL and PySpark |
| Bigger than memory | Yes, spills to disk | Yes, streaming engine and spilling group-bys | Yes, across a cluster |
| Bigger than one machine | No | No (open source) | Yes |
| Streaming | No | Batch streaming of large files | Yes, including real-time with state |
| Open table formats | Iceberg, Delta, DuckLake | Iceberg read and write, Delta | Iceberg and Delta, the reference engine |
| Shared access | New Quack server mode | In-process only | Spark Connect, multi-user clusters |
| AI coding agents | Agent mode in the CLI cuts output tokens | Clear errors, strict types | Feather speeds up the try-and-fix loop |
How to choose
- 1
Estimate your largest table
Under about 100 GB of Parquet, a single machine with DuckDB or Polars is usually enough. Into the terabytes, or growing fast, plan for Spark or a cloud warehouse.
- 2
Pick by the language your team thinks in
SQL people will be productive in DuckDB within an hour. Python developers will prefer Polars expressions.
- 3
Check where the job runs
If your company is on Databricks or Microsoft Fabric, Spark is already there and managed, and small jobs can still use DuckDB or Polars inside the same notebooks.
- 4
Need real-time? Use Spark
Neither DuckDB nor Polars processes event streams with state. Spark 4.3 now does it with latency around a tenth of a second.
- 5
Keep data in open formats
Store tables as Parquet or Iceberg so you can switch engines later without migrating data.
Check your understanding
1/3In our join-and-group-by test on 20 million rows, which engine was fastest?
DuckDB took 0.20 seconds, Polars 0.48, pandas 1.62 and Spark 1.83.
2/3When is Spark the right choice over DuckDB or Polars?
Spark's strength is scale and streaming; on small data its overhead dominates.
3/3What does Spark 4.3's Project Feather do?
In our test it cut a tiny query from 85 to 53 ms.
You got 0 out of 3.
Go deeper: DuckDB 2.0, Polars 2.1, Spark 4.3, the Polars cheat sheet and how to choose a database.
Sources: DuckDB 2.0.0 announcement (21 October 2026), Polars 2.1 release notes (25 October 2026) and Apache Spark 4.3.0 release notes (30 October 2026). Benchmarks are our own, run on 9 November 2026 on a 2-core, 7 GB cloud VM with synthetic data; they are indicative and not a substitute for testing your own workload.
Alessandro Ecclesie AgazziWritten by Alessandro Ecclesie Agazzi, analytics engineer in London. Updated 9 November 2026. About me