Top 3 this week
GM will pay $220 million to 36 attorneys general over selling driver data
A coalition of 35 states and Washington, D.C. says General Motors sold driving behavior and location data from its Smart Driver program to data brokers, who passed it to insurers, without telling drivers. GM must now get affirmative consent before collecting or sharing that data, limit what it collects, and run a privacy program with internal and outside assessments. Colorado says it is the first settlement under the Colorado Privacy Act.
Why it matters: Anyone building products on behavioral or location data should treat clear, opt-in consent as the expectation now, not a nice-to-have.
Apache Kafka discloses a batch of vulnerabilities, including forged login tokens
The Kafka project published a batch of security advisories on Oct 28. The most serious affects brokers that use OAuth token login without a JWKS endpoint (the URL Kafka uses to check token signatures): an attacker could forge a token and log in as any user. Versions 4.1.0 to 4.3.0 are affected, it is fixed in 4.2.2, 4.3.1 and 4.4.0, and setting sasl.oauthbearer.jwks.endpoint.url protects you without upgrading.
Why it matters: If you run Kafka with OAuth, check that setting today, and plan an upgrade for the rest of the batch.
Databricks makes Real-Time Mode for streaming pipelines generally available
Real-Time Mode processes streaming records continuously instead of in small batches, and it is now generally available in Spark Declarative Pipelines. Databricks says its own feature store reaches 200 millisecond latency for 99 percent of records from Kafka, and claims it is up to 92 percent faster than Apache Flink on its benchmark, which is a vendor claim.
Why it matters: Teams that run a separate engine like Flink just for low-latency work now have a real option to keep it on one platform.
Data Engineering
Snowflake's assistant can now set up Snowpipe ingestion from one request
Snowflake's AI assistant, CoCo, can now build a continuous ingestion pipeline (Snowpipe) from a plain-language request. It samples files in cloud storage, infers the schema, creates the pipeline and checks that data is loading, and it can generate Terraform or CloudFormation so the pipeline is managed as code. The feature is generally available.
Seven cloud data warehouses benchmarked side by side
Estuary and CorrDyn ran 244 queries against Snowflake, Databricks, BigQuery, Redshift, Microsoft Fabric, ClickHouse and MotherDuck, from 1 GB to 1 TB and up to 100 concurrent users, all on default settings. No single engine won, extra compute bought little speed at small data sizes, and concurrency and idle billing mattered as much as hourly price. The code and raw results are on GitHub.
Streaming and Real-Time
Kafka 4.3.2 fixes two potential data loss bugs
Kafka 4.3.2 is a bugfix release with about 60 fixes. The most important address potential data loss in tiered storage size-based retention and in time index truncation, and it updates several dependencies with published CVEs (Common Vulnerabilities and Exposures).
Flink Agents 0.4.0 adds sub-agents and tracing for AI agents in streaming jobs
Flink Agents runs AI agents inside Apache Flink streaming jobs. Version 0.4.0 adds sub-agents, per-request model routing, multimodal input and a trace log of model and tool calls. It is still a preview with breaking changes, and the project calls it the last planned release before 1.0.
Analytics and Semantic Modeling
AtScale: a semantic layer helped text-to-SQL accuracy more than switching models
AtScale tested seven large language models on 410 business questions, once against raw Postgres tables and once through its semantic layer. Switching models changed accuracy by up to 17.6 points, while adding the semantic layer added 20 to 31 points for every model. AtScale sells the semantic layer it tested, and the results are not yet independently verified.
How Okta governs the context its AI agents read, using dbt
Okta's AI agents were reading Snowflake semantic views that weren't tied to its governed dbt models. Okta moved agent metadata, context docs and semantic definitions into its dbt project, reviewed through pull requests, so every agent reads the same definitions.
Data Delivery
Looker 26.20 makes the new Explore the default and adds new chart types
Looker 26.20 makes the new Explore experience the default and brings bullet charts, a histogram chart and a new KPI chart to general availability, along with Gemini-generated Explore summaries. One breaking change: a value on the Conversational Analytics System Activity dashboard was renamed, which can break custom dashboards that filter on it.
Ask BigQuery questions in plain English from Google Sheets
Conversational analytics in Connected Sheets, which links Google Sheets to BigQuery, is now generally available. Users ask a question in plain language, and Gemini writes and runs the SQL, summarizes trends and inserts refreshable results into the workbook.
Data Governance and Privacy
FTC asks nine companies how personal data shapes prices and pay
The FTC ordered Delta, United, JetBlue, Lyft, Uber, DoorDash, Instacart, Target and Walmart to explain how personal data, such as loyalty activity or browsing behavior, is used to set individual prices. The ride-share and delivery companies must also explain how personal data feeds driver pay, and all nine have 45 days to respond.
France fines Fnac Darty 7 million euros over an app tracker used without consent
France's data protection regulator found that the retailer's app sent user identifiers and browsing activity to a third party for product recommendations and to train its recommendation model, without consent. The companies must show compliant consent collection within a month or pay 10,000 euros a day.
Data as a Product
TalentNeuron puts its labor market data on Snowflake Marketplace
TalentNeuron's job posting, skills and location data is now available as a governed, read-only share on Snowflake Marketplace. Customers can query it with SQL and join it with their own HR and finance data, with updates arriving automatically, alongside the company's existing APIs and feeds.
Outside engines can now read and write Snowflake-governed Iceberg tables
Spark, Trino, DuckDB and PyIceberg can now read and write externally managed Apache Iceberg tables through Snowflake Horizon's Iceberg REST catalog, with Snowflake's role-based access control applied. It makes it easier to share one governed table across tools and teams without copying it.
AI in the Data Stack
Airbyte releases an open source MCP server for managing pipelines from AI tools
Airbyte's MCP (Model Context Protocol) server lets AI clients such as Claude, ChatGPT and Cursor manage connections, diagnose failed syncs and manage custom connectors, and it is free on all plans. A separate context layer for Airbyte Cloud customers lets users run read-only queries on Snowflake or BigQuery from an AI client, with destructive actions requiring confirmation.
AI tools can connect to Databricks MCP servers without custom OAuth setup
Default OAuth apps for external AI agents are now generally available, so ChatGPT, Claude, Claude Code, Codex and Cursor can connect to Databricks MCP servers using built-in client IDs. Admins no longer have to create an OAuth app first.
Snowflake's document parser can now read charts, tables and diagrams
The AI_PARSE_DOCUMENT SQL function has a new AGENTIC mode, in public preview, that finds and classifies charts, diagrams and tables and returns summaries or structured data from them. Text, layout and extracted data come back from a single SQL call.
Platforms
Databricks adds cost controls for serverless notebooks, jobs and pipelines
Teams can now tag serverless compute to attribute spend to teams or projects, in Beta, and admins can limit which groups use serverless compute, now generally available. New Beta spend limits cap how far a job or pipeline can scale.
BigQuery can return query results in Apache Arrow format
In preview, BigQuery can now return results directly in Apache Arrow, the columnar format that pandas, Polars and DuckDB read natively. Google says this speeds up retrieving results into those tools.
ClickHouse's new join optimizer can reorder outer and anti joins
A new cost-based join optimizer, on by default in ClickHouse 26.10, searches every valid join order for queries with up to 12 tables. It can now reorder outer, semi and anti joins, not just inner joins, and the authors measured 25 to 55 percent faster runtimes on standard benchmarks.
Standards
Iceberg v4 will support statistics on derived values
The Iceberg community voted 15 to 0 to add stats-only fields to the v4 table spec. Files can then store statistics on derived values like partition fields or expressions such as lowercased strings, so query engines can skip more files when filtering on expressions.
Iceberg REST catalogs get an explicit no-credentials option
A vote passed to add none as an allowed value for the access delegation header in the Iceberg REST catalog spec. A client can now say explicitly that it does not want the catalog to hand out storage credentials.
Open Source
Apache Spark 4.3.0 is released
The Spark project released 4.3.0, a new feature release, after a release vote that passed without objections. The announcement points to the project's release notes for the feature list.
Germany's Sovereign Tech Agency funds PyArrow development
The German government-backed Sovereign Tech Agency is contracting about 1.9 full-time engineers through 2027 to work on PyArrow, the Python library for Apache Arrow. The work covers type annotations, the bug backlog, documentation, and tooling to help maintainers handle AI-generated contributions.
Data Security
Apache Pinot flaw lets users read tables they shouldn't
In Pinot's single-stage query engine, access control only checks the table in the FROM clause, so a user can use the lookup() function to read any dimension table and bypass row-level security. Versions 0.7.0 through 1.5.1 are affected, and the multi-stage engine is not.
Sony's So-net breach exposes about 2.6 million subscriber records
An attacker accessed the subscriber database of Sony's So-net internet service, exposing names, phone numbers, emails, birthdates and user IDs, plus passwords for about 54,000 email accounts. Sony says it has reset the exposed passwords.