Top 3 this week
DuckDB 2.0 is out as a long-term support release
DuckDB 2.0.0 is a long-term support release, supported until at least October 2027, and the team calls it their largest feature release so far. It adds a client-server mode with a new CONNECT statement for reaching remote databases, and a full VARIANT type for semi-structured data. DuckLake 1.1 shipped alongside it.
Why it matters: If you use DuckDB for local analytics or pipelines, this is the version to standardize on, and the new server mode opens it up to shared, multi-user setups.
EU Council agrees its position on GDPR and breach reporting changes
EU member states adopted their negotiating position on the Digital Omnibus, a package that simplifies data protection rules. It keeps the current GDPR definition of personal data, extends the breach notification window from 72 to 96 hours with a higher threshold, and adds safeguards for using sensitive data in AI development. This is a negotiating position, not law yet.
Why it matters: Teams that handle EU personal data should watch this closely, because breach response plans and AI training rules are both likely to change.
Tableau 2026.3 brings knowledge graphs and agents to dashboards
Tableau 2026.3 makes Tableau Knowledge generally available for Tableau Cloud+ and Tableau+, linking structured and unstructured data so agents can use it. The release also adds Tableau Agent in Dashboards and Tableau Data Apps in beta, plus a Tableau add-in for Excel.
Why it matters: If your organization runs Tableau, analysts will start asking questions of dashboards in plain English, so semantic definitions and governed data sources matter more than ever.
Data Engineering
dbt State now has a free tier
dbt State keeps track of build state across runs so unchanged models and tests can be skipped. It now has a free plan with 1,000 daily active target tables per month and no credit card required, and dbt platform users get 2,000 per month through March 2027.
Snowflake AIM puts its migration tools in one desktop app
AIM is a standalone desktop app, now in public preview, that walks a migration from connecting to the source database through code conversion, assessment, deployment and data validation. It uses AI to suggest fixes for code that does not convert cleanly.
Databricks tests agent-built pipelines, declarative vs. hand-written
Databricks had an AI agent build 528 pipelines both as plain PySpark and as Spark Declarative Pipelines. The declarative versions used about 34 times less cluster time and about half the code, but more AI tokens, and both styles produced silent logic errors at similar rates. It is a vendor benchmark, not a product launch.
Streaming and Real-Time
Debezium gets a command-line tool for change data capture pipelines
Debezium, the open source change data capture (CDC) project, introduced a CLI that scaffolds and validates pipeline configs and builds container images without Docker. It also manages environments and pipelines on Debezium Platform and can trigger actions like incremental snapshots.
Apache Beam can now run Debezium without Kafka
Apache Beam 2.76.0 includes a built-in Debezium connector that reads database changes directly inside a Beam pipeline, with no Kafka or Kafka Connect required. It supports MySQL, PostgreSQL, SQL Server and Db2 on any Beam runner, and it is still marked experimental.
Analytics and Semantic Modeling
Oracle announces a business semantic layer for AI agents
At Oracle AI World, Oracle announced a Business Semantic Layer for its Autonomous AI Lakehouse that defines reusable metrics, hierarchies and KPIs for natural-language querying. Oracle says it will arrive within 12 months, alongside an AI data catalog and full Iceberg table support.
Gartner puts AI context platforms on its 2027 trends list
Gartner's top strategic technology trends for 2027 include AI context platforms, which give agents the business meaning behind data. Gartner predicts that prioritizing semantics in AI-ready data could raise agent accuracy by up to 80 percent.
Data Delivery
ThoughtSpot launches AgentSpot for building and sharing AI agents
AgentSpot lets ThoughtSpot users build AI agents from plain-English prompts and share them across the company, running on ThoughtSpot's governed semantic layer. There is a free tier with no user limit and 25 credits a week.
Sigma adds export controls and metric-aware KPIs
Sigma's assistant now respects a governed metric's timeline and comparison settings when it builds a KPI. A new beta toggle restricts where and what users can export, and Redshift Serverless connections are in beta.
Data Governance and Privacy
Judge orders Meta to pay New Mexico $25 billion over privacy claims
A New Mexico judge ordered Meta to pay $25 billion after a jury found 43 million violations tied to Facebook's privacy statements following Cambridge Analytica. Meta says it will appeal.
Varonis sensitivity labels now flow into the Collibra catalog
Varonis and Collibra integrated so sensitive-data labels sync continuously into the Collibra catalog. Data stewards can see whether a dataset is fit for AI training or analytics, and regulated personal, health and financial data gets flagged in Collibra workflows.
Data as a Product
EU Data Act: trade secrets no longer block data sharing by default
Under the EU Data Act, data that counts as a trade secret still has to be shared with users and third parties, with agreed confidentiality measures. A company can refuse only with evidence that disclosure is highly likely to cause serious economic harm, which matters for anyone publishing data products in the EU.
AI in the Data Stack
Databricks jobs can now run AI agent tasks
Genie Code tasks are now generally available in Lakeflow Jobs, so a scheduled job can run an AI prompt as one of its steps. The task can read earlier task outputs, call tools and return a link to its conversation.
Coding agents can now drive Fabric notebooks
Microsoft released the Fabric Notebook Toolkit in preview, a command-line tool that lets coding agents such as Claude Code, GitHub Copilot CLI and Codex run, inspect and edit cells in live Fabric notebooks. It returns output, Spark job status and logs in a format agents can read.
Snowflake MCP servers can offer a coding agent
Snowflake-managed MCP (Model Context Protocol) servers can now include a coding agent tool, which gives each conversation its own managed coding sandbox. Calls can pass a thread ID to continue the same conversation.
Platforms
Snowflake Online Feature Store is generally available
Snowflake's Online Feature Store now serves the latest feature values with low latency for real-time machine learning, such as fraud detection and recommendations. It runs in AWS and Azure commercial regions, not Google Cloud.
Databricks serverless compute can now cache DataFrames
You can now use df.cache() or df.persist() on serverless compute to reuse expensive results for the rest of a session. It is in public preview, needs serverless environment version 5 or later, and is on by default for most workspaces.
Fabric Migration Assistant now handles Teradata
Fabric's Migration Assistant for Data Warehouse can now convert Teradata tables, views, procedures and macros to Fabric T-SQL, in preview. It flags scripts that need manual fixes, and moving the data itself is a separate step.
Amazon OpenSearch can build AI search indexes straight from S3
OpenSearch intelligent ingestion is a managed, serverless feature that parses, chunks, embeds and indexes PDFs, Word files and images from Amazon S3 for retrieval-augmented generation (RAG). AWS estimates about $0.40 per 1,000 PDF pages.
Standards
Iceberg v4 votes open on table constraints and simpler file stats
The Iceberg community opened a vote to add check, unique and primary-key constraints to the v4 table spec, with primary keys replacing identifier fields. A same-day vote proposes replacing per-column tight bounds with one file-level stats flag, and neither had a final result posted by Oct 25.
DuckLake 1.1 ships with a new version of its spec
DuckLake 1.1 adds nested schemas, epoch partition transforms and exact string statistics under a new 1.1 version of the lakehouse format spec. It can still read and write 1.0 catalogs, and upgrading is opt-in.
Open Source
Polars 2.1 adds recursive SQL, Iceberg writes and encrypted Parquet
Polars 2.1.0 adds SQL recursive CTEs, creating tables in attached Iceberg catalogs, and reading and writing encrypted Parquet, plus speedups for joins and large datasets. It drops Python 3.10 support.
pandas 3.1 speeds up read_csv and adds a select method
pandas 3.1.0 adds DataFrame.select() for column expressions and reports read_csv running 2 to 3 times faster, with large local files read in parallel by default. Check the behavior changes before upgrading.
Data Security
Apache Pulsar fixes 14 vulnerabilities, including a critical auth bypass
The Pulsar project disclosed 14 vulnerabilities fixed in 4.0.14, 4.2.5 and 5.0.0. The critical one lets unverified tokens get any role, including superuser, under a specific authorization setup, and several others allow access across tenants.
Apache CouchDB 3.5.3 fixes six vulnerabilities
A critical flaw in CouchDB 3.5.2 and earlier lets a server admin run shell commands on the host, and five more issues include an authentication bypass and credential leaks through replication URLs. The fix is upgrading to 3.5.3.
PingCAP breach exposed TiDB Cloud customer details
Attackers used stolen service-account credentials to reach PingCAP systems from September 9 to October 7 and took source code plus TiDB Cloud user, contract and billing details. PingCAP says data stored in customers' TiDB databases was not affected.