--- title: Rivian expands the Delta Lake ecosystem with Delta-Go description: Rivian open-sourced Delta-Go, a Go connector that writes directly into Delta Lake tables, built to handle real-time ingestion of PCAP data from over 80,000... canonical: https://daily.dev/posts/rivian-expands-the-delta-lake-ecosystem-with-delta-go-0zhkvy0td twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Rivian expands the Delta Lake ecosystem with Delta-Go | daily.dev og:description: Rivian open-sourced Delta-Go, a Go connector that writes directly into Delta Lake tables, built to handle real-time ingestion of PCAP data from over 80,000... og:url: https://daily.dev/posts/rivian-expands-the-delta-lake-ecosystem-with-delta-go-0zhkvy0td og:image: https://api.daily.dev/og/posts/0zHkVY0tD.png og:image:alt: Rivian expands the Delta Lake ecosystem with Delta-Go og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Rivian expands the Delta Lake ecosystem with Delta-Go **[Delta Lake Blog](https://daily.dev/sources/delta-lake)** · 5 min read · 0 upvotes · 0 comments ## Summary Rivian open-sourced Delta-Go, a Go connector that writes directly into Delta Lake tables, built to handle real-time ingestion of PCAP data from over 80,000 connected vehicles at rates of 9.8 million records per second. The connector replaced an earlier architecture that duplicated data into Parquet and Delta formats and required 20 nonstop Spark workers, cutting that overhead and saving an estimated $500 per day. A DynamoDB-backed logstore was added to prevent race conditions between simultaneous writes from Spark and Delta-Go, and the system runs as a Go application on Kubernetes consuming SQS notifications, achieving 10 commits per second to the Delta table. Rivian is inviting community contributions to expand the connector's capabilities beyond its current business-specific scope. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### How does Rivian's Delta-Go connector avoid race conditions when Spark and Go both write to the same Delta Lake table? It uses a DynamoDB-backed logstore shared between Spark and Delta-Go, so both engines write through the same backend and coordinate table version updates, preventing simultaneous writes from overwriting each other's commits. This compatibility layer let Rivian keep using Spark Optimize alongside the new Go-based ingestion path without conflicts. _Teams building multi-engine data pipelines can track patterns like this shared-logstore approach on daily.dev._ ### Why did Rivian replace its Parquet-to-Delta ingestion pipeline with a direct Delta Lake writer in Go? Writing PCAP data first into Parquet files and then converting them into Delta tables via Spark Auto Loader caused storage redundancy and constant reprocessing, requiring 20 nonstop Spark workers. Building a Go connector that writes directly into Delta Lake eliminated dual-format storage and the conversion overhead, saving an estimated $500 per day. _Engineers weighing ingestion architectures for high-volume streaming data can follow similar case studies on daily.dev._ ### How many commits per second can the Delta-Go connector write to a Delta Lake table? The Delta-Go connector achieved 10 commits per second when writing PCAP data directly into a Delta Lake table, handling ingestion from over 80,000 connected vehicles at up to 9.8 million records per second. Rivian later optimized the code by consolidating transactions into larger logs to further improve downstream processing efficiency. _Anyone benchmarking real-time ingestion throughput can find comparable engineering writeups on daily.dev._ ## Similar posts on daily.dev - [Delta Kernel \(Rust\) and the future of the Lakehouse.](https://daily.dev/posts/delta-kernel-rust-and-the-future-of-the-lakehouse--twurulaae) · Confessions of a Data Guy · 0 upvotes · 0 comments - [Announcing General Availability of Zerobus Ingest, part of Lakeflow Connect](https://daily.dev/posts/announcing-general-availability-of-zerobus-ingest-part-of-lakeflow-connect-0rtyz2hni) · databricks · 0 upvotes · 0 comments - [650GB of Data \(Delta Lake on S3\). Polars vs DuckDB vs Daft vs Spark.](https://daily.dev/posts/650gb-of-data-delta-lake-on-s3-polars-vs-duckdb-vs-daft-vs-spark--bbkmmear9) · Hacker News · 1 upvotes · 0 comments - [ClickHouse is data lake ready](https://daily.dev/posts/clickhouse-is-data-lake-ready-yi7rw0mlt) · ClickHouse · 2 upvotes · 0 comments --- Tags: [#kubernetes](https://daily.dev/tags/kubernetes), [#golang](https://daily.dev/tags/golang), [#data-engineering](https://daily.dev/tags/data-engineering), [#apache-spark](https://daily.dev/tags/apache-spark) [View this post on daily.dev](https://daily.dev/posts/rivian-expands-the-delta-lake-ecosystem-with-delta-go-0zhkvy0td) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Rivian expands the Delta Lake ecosystem with Delta-Go","url":"https://daily.dev/posts/rivian-expands-the-delta-lake-ecosystem-with-delta-go-0zhkvy0td","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/rivian-expands-the-delta-lake-ecosystem-with-delta-go-0zhkvy0td"},"datePublished":"2026-08-16T08:53:52.215Z","dateModified":"2026-08-16T08:56:25.947Z","description":"Rivian open-sourced Delta-Go, a Go connector that writes directly into Delta Lake tables, built to handle real-time ingestion of PCAP data from over 80,000...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/17196c89f3d2426ac60828ff828f1678?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/17196c89f3d2426ac60828ff828f1678?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Delta Lake Blog","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Delta Lake Blog","logo":"https://media.daily.dev/image/upload/s--7f2q9Asf--/f_auto,q_auto/v1786870393/logos/delta-lake?_a=BAMAMicg0","url":"https://daily.dev/sources/delta-lake"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/rivian-expands-the-delta-lake-ecosystem-with-delta-go-0zhkvy0td","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"kubernetes,golang,data-engineering,apache-spark","timeRequired":"PT5M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Delta Lake Blog","item":"https://daily.dev/sources/delta-lake"},{"@type":"ListItem","position":3,"name":"Rivian expands the Delta Lake ecosystem with Delta-Go"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/rivian-expands-the-delta-lake-ecosystem-with-delta-go-0zhkvy0td#faq","mainEntity":[{"@type":"Question","name":"How does Rivian's Delta-Go connector avoid race conditions when Spark and Go both write to the same Delta Lake table?","acceptedAnswer":{"@type":"Answer","text":"It uses a DynamoDB-backed logstore shared between Spark and Delta-Go, so both engines write through the same backend and coordinate table version updates, preventing simultaneous writes from overwriting each other's commits. This compatibility layer let Rivian keep using Spark Optimize alongside the new Go-based ingestion path without conflicts. Teams building multi-engine data pipelines can track patterns like this shared-logstore approach on daily.dev."}},{"@type":"Question","name":"Why did Rivian replace its Parquet-to-Delta ingestion pipeline with a direct Delta Lake writer in Go?","acceptedAnswer":{"@type":"Answer","text":"Writing PCAP data first into Parquet files and then converting them into Delta tables via Spark Auto Loader caused storage redundancy and constant reprocessing, requiring 20 nonstop Spark workers. Building a Go connector that writes directly into Delta Lake eliminated dual-format storage and the conversion overhead, saving an estimated $500 per day. Engineers weighing ingestion architectures for high-volume streaming data can follow similar case studies on daily.dev."}},{"@type":"Question","name":"How many commits per second can the Delta-Go connector write to a Delta Lake table?","acceptedAnswer":{"@type":"Answer","text":"The Delta-Go connector achieved 10 commits per second when writing PCAP data directly into a Delta Lake table, handling ingestion from over 80,000 connected vehicles at up to 9.8 million records per second. Rivian later optimized the code by consolidating transactions into larger logs to further improve downstream processing efficiency. Anyone benchmarking real-time ingestion throughput can find comparable engineering writeups on daily.dev."}}]} ```