<!-- mobian-agent-page publisher="dailydev" canonical="https://daily.dev/posts/what-s-new-in-managed-service-for-apache-spark-4-0-baavowpu7" -->

---
title: What’s new in Managed Service for Apache Spark 4.0
description: Google Cloud has made Managed Service for Apache Spark 4.0 generally available for both serverless and managed cluster deployments. The release standardizes on...
canonical: https://daily.dev/posts/what-s-new-in-managed-service-for-apache-spark-4-0-baavowpu7
twitter:card: summary_large_image
twitter:site: @dailydotdev
og:type: website
og:site_name: daily.dev
og:title: What’s new in Managed Service for Apache Spark 4.0 | daily.dev
og:description: Google Cloud has made Managed Service for Apache Spark 4.0 generally available for both serverless and managed cluster deployments. The release standardizes on...
og:url: https://daily.dev/posts/what-s-new-in-managed-service-for-apache-spark-4-0-baavowpu7
og:image: https://api.daily.dev/og/posts/bAAVowPu7.png
og:image:alt: What’s new in Managed Service for Apache Spark 4.0
og:image:width: 1200
og:image:height: 630
og:locale: en
---

> ## Documentation Index
> Fetch the complete documentation index at: https://daily.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# What’s new in Managed Service for Apache Spark 4.0

**[Google Cloud](https://daily.dev/sources/gcp)** · 10 min read · 0 upvotes · 0 comments

## Summary

Google Cloud has made Managed Service for Apache Spark 4.0 generally available for both serverless and managed cluster deployments. The release standardizes on Apache Spark 4.2, Java 25, Python 3.13 with Pixi, and Apache Iceberg 1.12. It adds a redesigned code-first console with AI-driven troubleshooting, a Data Agent Kit for building Spark pipelines from VS Code, Claude Code, and Antigravity, and faster serverless startups on Gen4 infrastructure. The Lightning Engine vectorized execution engine now delivers up to 3.25x the price-performance over the leading high-speed Spark alternative and up to 4.9x faster performance than open-source Spark. Serverless deployments move to unified DCU quotas and automatically attach to the Lakehouse runtime catalog instead of a legacy Hive Metastore.

## Full article

daily.dev links to this article rather than hosting it. Read it at the original source: <https://cloud.google.com/blog/products/data-analytics/whats-new-in-managed-service-for-apache-spark-4-0>

## Questions this post answers

### What version of Apache Spark does Google Cloud's Managed Service for Apache Spark 4.0 runtime use?

It runs on Apache Spark 4.2, alongside Java 25, Scala 2.13, Python 3.13 with Pixi for dependency management, Apache Iceberg 1.12, and Delta Lake 4.4. This runtime is standardized across both serverless and managed cluster deployments, so PySpark, Scala, and Spark SQL workloads can move between the two without version mismatches.

_Teams planning a Spark runtime upgrade can track version changes like these on daily.dev._

### How much faster is Lightning Engine than regular open-source Apache Spark?

Lightning Engine delivers up to 4.9x faster performance than running open-source Apache Spark with zero code changes, and up to 3.25x the price-performance over the leading high-speed Spark alternative. It also speeds up Apache Iceberg file scans by up to 50% through intelligent caching and accelerates BigQuery reads by up to 40% using Apache Arrow streams.

_Evaluating Spark execution engines gets easier when daily.dev surfaces benchmark claims like this one._

### What is the Data Agent Kit for Managed Service for Apache Spark?

It is a setup tool that configures Model Context Protocol (MCP) servers, agent skills, and API access so AI agents can build, tune, and run Apache Spark pipelines directly from an IDE. It supports VS Code, Claude Code, and Antigravity, letting data engineers generate and test Spark workloads on the 4.0 runtime without leaving their editor.

_Developers wiring MCP into their coding workflow can follow tooling updates like this on daily.dev._

## Similar posts on daily.dev

- [Enhancements to Managed Service for Apache Spark clusters](https://daily.dev/posts/enhancements-to-managed-service-for-apache-spark-clusters-vd2cxc41m) · Google Cloud · 0 upvotes · 0 comments
- [Serverless Managed Service for Apache Spark runtime 3.0 features](https://daily.dev/posts/serverless-managed-service-for-apache-spark-runtime-3-0-features-wjxmpremn) · Google Cloud · 0 upvotes · 0 comments
- [Lighting Engine for Apache Spark performance deep dive](https://daily.dev/posts/lighting-engine-for-apache-spark-performance-deep-dive-uqwn7cruj) · Google Cloud · 0 upvotes · 0 comments
- [Amazon EMR Serverless now supports Apache Spark 4.0.1 \(preview\)](https://daily.dev/posts/amazon-emr-serverless-now-supports-apache-spark-4-0-1-preview--mckowuiut) · AWS · 0 upvotes · 0 comments
- [Develop and deploy Serverless JARs](https://daily.dev/posts/develop-and-deploy-serverless-jars-spy7rgvds) · databricks · 0 upvotes · 0 comments

---

Tags: [#gcp](https://daily.dev/tags/gcp), [#mcp](https://daily.dev/tags/mcp), [#data-engineering](https://daily.dev/tags/data-engineering), [#apache-spark](https://daily.dev/tags/apache-spark), [#apache-iceberg](https://daily.dev/tags/apache-iceberg)

[View this post on daily.dev](https://daily.dev/posts/what-s-new-in-managed-service-for-apache-spark-4-0-baavowpu7)

```json
{"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]}
{"@context":"https://schema.org","@type":"TechArticle","headline":"What’s new in Managed Service for Apache Spark 4.0","url":"https://daily.dev/posts/what-s-new-in-managed-service-for-apache-spark-4-0-baavowpu7","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/what-s-new-in-managed-service-for-apache-spark-4-0-baavowpu7"},"datePublished":"2026-11-20T17:08:51.668Z","dateModified":"2026-11-20T17:09:19.176Z","description":"Google Cloud has made Managed Service for Apache Spark 4.0 generally available for both serverless and managed cluster deployments. The release standardizes on...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/d9c0622b5f8757632d94ccd5f5445d1c?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/d9c0622b5f8757632d94ccd5f5445d1c?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Google Cloud","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Google Cloud","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/gcp","url":"https://daily.dev/sources/gcp"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/what-s-new-in-managed-service-for-apache-spark-4-0-baavowpu7","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"gcp,mcp,data-engineering,apache-spark,apache-iceberg","timeRequired":"PT10M"}
{"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Google Cloud","item":"https://daily.dev/sources/gcp"},{"@type":"ListItem","position":3,"name":"What’s new in Managed Service for Apache Spark 4.0"}]}
{"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/what-s-new-in-managed-service-for-apache-spark-4-0-baavowpu7#faq","mainEntity":[{"@type":"Question","name":"What version of Apache Spark does Google Cloud's Managed Service for Apache Spark 4.0 runtime use?","acceptedAnswer":{"@type":"Answer","text":"It runs on Apache Spark 4.2, alongside Java 25, Scala 2.13, Python 3.13 with Pixi for dependency management, Apache Iceberg 1.12, and Delta Lake 4.4. This runtime is standardized across both serverless and managed cluster deployments, so PySpark, Scala, and Spark SQL workloads can move between the two without version mismatches. Teams planning a Spark runtime upgrade can track version changes like these on daily.dev."}},{"@type":"Question","name":"How much faster is Lightning Engine than regular open-source Apache Spark?","acceptedAnswer":{"@type":"Answer","text":"Lightning Engine delivers up to 4.9x faster performance than running open-source Apache Spark with zero code changes, and up to 3.25x the price-performance over the leading high-speed Spark alternative. It also speeds up Apache Iceberg file scans by up to 50% through intelligent caching and accelerates BigQuery reads by up to 40% using Apache Arrow streams. Evaluating Spark execution engines gets easier when daily.dev surfaces benchmark claims like this one."}},{"@type":"Question","name":"What is the Data Agent Kit for Managed Service for Apache Spark?","acceptedAnswer":{"@type":"Answer","text":"It is a setup tool that configures Model Context Protocol (MCP) servers, agent skills, and API access so AI agents can build, tune, and run Apache Spark pipelines directly from an IDE. It supports VS Code, Claude Code, and Antigravity, letting data engineers generate and test Spark workloads on the 4.0 runtime without leaving their editor. Developers wiring MCP into their coding workflow can follow tooling updates like this on daily.dev."}}]}
```

