Top 3 this week
Apache Airflow 3.4.0 adds Task Loops and Dags in other languages, and closes 18 vulnerabilities
Airflow 3.4.0 adds Task Loops, which repeat a group of tasks a number of times decided at runtime, plus experimental support for Dags (workflows) written in YAML, TypeScript, Java or Go. Psycopg3 becomes the default Postgres driver and Python 3.10 support is dropped. On Nov 9 the project disclosed 18 CVEs (published security vulnerabilities) fixed by this release, including one that let anyone who could write to the Celery message broker run code on every Celery worker, and one that left connection and variable secrets in readable audit logs.
Why it matters: If you run Airflow, the security fixes alone make 3.4.0 an upgrade to schedule now, so test your Dags against it this month.
Confluent makes Streamhouse and Lightning Tables generally available on AWS
Confluent's fourth-quarter release makes its Streamhouse architecture and Lightning Tables generally available on AWS. The idea is that a Kafka topic and an analytics table are kept as one copy of the data, so the same stream can feed real-time applications and be queried as a table without a separate copy job. Confluent is pitching it as the way to give AI agents current business context.
Why it matters: If your team copies Kafka topics into a lake or warehouse just to query them, this is worth comparing against the pipelines you maintain today.
Databricks renames Delta Sharing to OpenSharing and expands what can be shared
Delta Sharing, the open protocol for sharing live tables across companies and platforms without copying them, is now called OpenSharing. Alongside the rename, Databricks made SecureConnect generally available, put sharing of Genie agents (its natural-language analytics assistant) in Beta, and previewed sharing for Lakebase, its Postgres database. The company says the protocol now covers more than Delta tables.
Why it matters: Teams that send partners nightly file drops or API extracts should look at whether a shared live table could replace them.
Data Engineering
Apache Spark 4.3 arrives in Databricks Runtime 19
Spark 4.3 is now available on Databricks. Highlights include stateful processing in Real-Time Mode, built-in handling of slowly changing dimensions (SCD Type 2, which keeps the history of changed records) through Auto CDC, and new SQL features such as JSON_TABLE and ASOF JOIN for matching records to the nearest earlier timestamp.
dltHub makes pipeline observability generally available
dltHub, the company behind the open-source dlt Python library for loading data, released pipeline observability as a generally available feature. It tracks pipeline runs, loads and schema changes so teams can see what loaded, when, and what changed without building their own monitoring.
JetBrains builds cross-database queries into DataGrip with DuckDB
JetBrains embedded DuckDB in DataGrip, its database IDE (integrated development environment), so users can join tables from different database connections in one query. The team also evaluated Apache Calcite and H2 and picked DuckDB because it was simplest to integrate and supports pushing filters down to the source databases.
Streaming and Real-Time
Debezium and Databricks Zerobus push change data straight into Unity Catalog
Databricks shows how to send change data capture (CDC) events from Debezium, the open-source tool that reads database change logs, directly into Unity Catalog tables through its Zerobus ingest service. The approach skips a Kafka cluster in the middle for teams that only need database changes landed in the lakehouse.
ClickHouse makes its MongoDB CDC connector generally available
ClickPipes, ClickHouse Cloud's managed ingestion service, now has a generally available connector that continuously replicates MongoDB changes into ClickHouse. It lets teams run fast analytics on operational MongoDB data without writing their own sync jobs.
Analytics and Semantic Modeling
AtScale connects its semantic layer to Grok through MCP
AtScale released a plugin that exposes its semantic layer to xAI's Grok through MCP (Model Context Protocol, a standard way for AI assistants to call outside tools). Questions asked in Grok are answered using the business metric definitions already set up in AtScale, rather than the model guessing at raw tables.
ClickHouse 26.10 adds non-equality joins and a pipeline operator
JOINs in ClickHouse now accept any ON condition, including ranges and other non-equality comparisons, and there is experimental LATERAL JOIN support. A new | operator chains functions into a readable pipeline, large IN subqueries can spill to disk, and SQLite is supported as an input and output format.
Data Delivery
Snowflake previews Dashboards and HTML Reports
Snowflake put Dashboards and HTML Reports into public preview, so results built in Snowflake can be shared as dashboards or self-contained reports without a separate BI tool. It was announced alongside a set of AI agent features at Snowflake's November event.
Streamlit 1.67.0 adds a grid layout and an experimental agent endpoint
Streamlit, the Python library for building data apps, added st.grid for laying out elements in rows and columns. It also added an experimental agent API with an MCP endpoint, so an AI assistant can interact with a Streamlit app.
Data Governance and Privacy
Seoul court rejects Coupang's challenge to a $315 million privacy fine
The Seoul Administrative Court rejected Coupang's lawsuit to nullify a 423 billion won (about $315 million) fine from South Korea's Personal Information Protection Commission. The fine, the commission's largest ever on one company, followed a breach that exposed names, phone numbers and delivery details of more than 37 million users, which Coupang reported late.
France's privacy regulator sanctions ten local governments over weak cybersecurity
CNIL, France's data protection authority, fined ten French local authorities between 3,000 and 20,000 euros each for failing to secure the personal data they hold. The penalties are small, but they show regulators treating basic security gaps as privacy violations even in the public sector.
Data as a Product
Snowflake launches an AI Data Package with more than 60 datasets
Snowflake announced an AI Data Package that bundles more than 60 third-party datasets for use by AI agents running in Snowflake, in public preview. It is part of a wider push to sell ready-to-use external data alongside the platform rather than leaving teams to source it themselves.
AI in the Data Stack
BigQuery previews AI.AGG for summarizing groups of rows with a model
AI.AGG is a new BigQuery aggregate function, in preview, that passes a group of rows to a Gemini model and returns one result, such as a summary of all reviews for a product. It works in a GROUP BY like SUM or COUNT, which brings model-based summarization into ordinary SQL.
MotherDuck adds an agent chat that can query and explain your data
MotherDuck, the cloud service built on DuckDB, added Agent Chat, which answers questions by writing and running queries against your MotherDuck databases. It shows the SQL it ran so analysts can check the work.
Platforms
Amazon S3 Tables can now migrate Hive tables to Iceberg for you
S3 Tables, AWS's managed Apache Iceberg table storage, now offers a managed migration that converts existing Hive tables to Iceberg. It targets teams with older Hive-based data lakes who want Iceberg features such as time travel and schema evolution without a hand-built migration.
AWS Glue Data Catalog adds vended credentials for Iceberg tables
The Glue Data Catalog can now hand out short-lived, table-scoped storage credentials to engines that read Iceberg tables. Engines no longer need broad S3 permissions of their own, which makes it easier to govern access from tools outside AWS.
Standards
Iceberg approves default value expressions for the v4 table format
Apache Iceberg is an open table format for large analytic datasets. The community voted to let a column's write default be an expression, written in Iceberg's expression spec, rather than only a fixed value. The change is part of the upcoming v4 specification.
Hudi-rs 0.6.0 writes Hudi tables natively from Rust and Python
Hudi-rs, the native Rust implementation of the Apache Hudi table format, can now create tables and run appends, upserts, updates, deletes and overwrites without a JVM (Java Virtual Machine). It also adds compaction for Merge-on-Read tables and reports full-table Python reads 3 to 6 times faster.
Open Source
Apache Arrow 26.0.1 fixes a Brotli vulnerability in PyArrow wheels
Arrow is the in-memory columnar format used by pandas, Polars, DuckDB and others. This patch release updates the bundled Brotli compression library to fix a vulnerability in PyArrow wheels and R binaries, and fixes Parquet treating an unknown compression codec as uncompressed and an out-of-bounds read on invalid input.
Iceberg Rust 0.11.0 adds encryption and more v3 features
The native Rust library for Iceberg tables adds write support for client-side encryption. It also implements more of the v3 spec, including row lineage, nanosecond timestamps and delete vectors.
Data Security
ZooKeeper fixes a bug that let clients claim superuser identity
Apache ZooKeeper, the coordination service behind many Kafka, HBase and Solr deployments, disclosed five vulnerabilities fixed in 3.8.8 and 3.9.7. The most serious (rated 8.6 out of 10) let an authenticated SASL client claim any identity, including a configured superuser, and take full control of ZooKeeper's data. Another let deeply nested nodes crash the server in a way that survives restarts.
DigiCert says an exposed archive let outsiders read customer identity documents
An internal DigiCert archive was accidentally left reachable from the internet after a network move. Attackers used its self-registration feature, which did not require multi-factor authentication, to read business customers' contact details and, in some cases, driver's licenses and passports. DigiCert says its certificate systems were not affected.
unixODBC driver manager has unpatched memory bugs
unixODBC is the ODBC (Open Database Connectivity) driver manager many Linux database clients and BI tools rely on. A researcher reported heap buffer overflows, an out-of-bounds copy and race conditions in version 2.3.14 and earlier that could cause crashes or possibly code execution. No fix or CVE has been published yet.