Last week was all about Snowflake Expedition. This week, 4 to 9 November 2026, the other platforms took their turn. The common thread: open table formats, business context for AI agents, and Postgres turning up everywhere. Last week's edition covers 28 October to 3 November.
Databricks
Platform
New this week- Apache Spark 4.3, which Databricks says is available in Runtime 19, with Real-Time Mode for stateful streams and SCD Type 2 Auto CDC
- Lakebase Postgres generally available on Google Cloud in five regions, with scale to zero, branching and a 99.99% SLA
- AI Function tasks for jobs (beta): run ai_parse_document, ai_extract or ai_classify on new files or rows
- Claude Code and Codex now work in the Databricks SSH tunnel (CLI 1.23.0 or later)
Spark 4.3 is the headline; read what's new in Spark 4.3.
Governance and BI
Beta and GA- External knowledge sources (beta): pull Power BI semantic model metadata into Databricks so Genie uses the same measure definitions
- Unity Catalog wikis (beta): document what a table means, with comments and suggested edits
- Metric view listings in Databricks Marketplace are GA
- Ingress policies can now allow Power BI, Tableau Cloud and dbt Cloud without tracking IP addresses
Good news for teams that use Databricks and Power BI together.
Plan ahead
Check your jobs- Kimi K3.1 Pro is now available on Unity Gateway in us-east-1
- DeepSeek V4 Flash (0731) is retired; move to DeepSeek V4.1 Flash (1017)
- From January 2027, AI Functions in SQL will need Databricks Runtime 17.3 LTS or later
Update endpoints and runtimes before they break.
Google Cloud
Google's 4 November update was the biggest of the week. BigQuery Graph is generally available, cross-cloud interconnect and caching for AWS and Azure is GA, AlloyDB can query Iceberg tables directly (preview), and Knowledge Catalog learns business context from documents and reuses semantic models from Looker, dbt, Databricks and Snowflake. Full details in BigQuery Graph goes GA: Google's November data cloud update.
AWS, Snowflake and the rest
4 to 9 November
AWS S3 Tables migrates Hive to Iceberg
Validates, converts and copies Glue-registered Parquet tables while production runs; pay only for copied data. Our Iceberg migration guide compares it with Spark's procedures.
Percona Search for MongoDB GA
Free full-text, vector and hybrid search with embeddings generated locally.
EDB to acquire Altinity
Joining PostgreSQL transactions with ClickHouse analytics, with a shared Iceberg catalog on the roadmap.
Confluent Streamhouse
Lightning Tables, now GA on AWS, turn Kafka topics into a read-only SQL and API layer; a Real-Time Context Engine feeds agents through MCP.
Redis Iris expands
Agent routing, reusable playbooks, governed memory and a context gateway are GA.
Snowflake after Expedition
Snowflake Practitioner, free role-based labs for analysts, engineers and admins; Q3 results due 2 December.
AI news that matters for data teams
- GPT-6 Cyber (5 November): OpenAI's security model wrote working exploits for 948 Linux kernel flaws and scored 48.5% on ExploitGym. Access is for vetted defenders only. See GPT-6 Cyber, explained.
- ChatGPT Enterprise in AWS Marketplace (9 November): companies can now buy ChatGPT Enterprise through their AWS account.
- Free AI models changed: Gemini Free now picks the model for you. See the best free AI models in November 2026.
The week ahead
- 13 November, Dublin: EU leaders hold their first dedicated summit on AI, focused on competitiveness, AI sovereignty and international cooperation. Mistral's CEO Arthur Mensch and Geoffrey Hinton are expected to speak.
- 17 to 20 November, San Francisco and online: Microsoft Ignite. Keynote at 18:00 UK time on the 17th. Our Ignite preview explains what to watch for Power BI and Fabric.
Check your understanding
1/3What can Databricks' new external knowledge sources beta read?
Genie can reuse the measure definitions you already maintain in Power BI.
2/3Which AWS service now migrates Hive tables to Iceberg for you?
It supports Glue-registered Parquet tables in the same account and region.
3/3What should you do about DeepSeek V4 Flash (0731) on Databricks?
The old endpoint is retired; update any jobs that call it.
You got 0 out of 3.
More: last week's data platform news, DuckDB vs Polars vs Spark and ChatGPT vs Claude vs Gemini for data analysis.
Sources: Databricks platform release notes (1 to 9 November 2026) and blog posts on Spark 4.3 and Lakebase on Google Cloud; Google Cloud blog (4 November 2026); AWS What's New (4 November 2026); Solutions Review data management news (6 November 2026) for Confluent, EDB, Percona and Redis; Enterprise Times (9 November 2026) for Snowflake; OpenAI (5 and 9 November 2026); Brussels Times (4 November 2026) on the Dublin summit.
Alessandro Ecclesie AgazziWritten by Alessandro Ecclesie Agazzi, analytics engineer in London. Updated 9 November 2026. About me