--- title: Jaeger hits 8.6× compression with ClickHouse, BigQuery's... description: A data engineering roundup covering several notable developments: Jaeger v2.18.0 ships ClickHouse as an alpha trace storage backend with 8.6× compression and... canonical: https://daily.dev/posts/jaeger-hits-8-6-compression-with-clickhouse-bigquery-s-hidden-slot-costs-explained-vta4rjj1g twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Jaeger hits 8.6× compression with ClickHouse, BigQuery's hidden slot costs explained | daily.dev og:description: A data engineering roundup covering several notable developments: Jaeger v2.18.0 ships ClickHouse as an alpha trace storage backend with 8.6× compression and... og:url: https://daily.dev/posts/jaeger-hits-8-6-compression-with-clickhouse-bigquery-s-hidden-slot-costs-explained-vta4rjj1g og:image: https://api.daily.dev/og/posts/Vta4rjj1g.png og:image:alt: Jaeger hits 8.6× compression with ClickHouse, BigQuery's hidden slot costs explained og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Jaeger hits 8.6× compression with ClickHouse, BigQuery's hidden slot costs explained **[Databases Digest](https://daily.dev/sources/databases_digest)** · 4 min read · 0 upvotes · 0 comments ## Summary A data engineering roundup covering several notable developments: Jaeger v2.18.0 ships ClickHouse as an alpha trace storage backend with 8.6× compression and 50k+ spans/sec ingestion; two BigQuery deep-dives explain slot-based billing and columnar internals rooted in the Dremel paper; Datasette 1.0a30 adds a keyboard-driven jump menu with plugin hooks; and a thorough guide covers probabilistic data structures (HyperLogLog, Theta sketches, KLL quantiles) for approximate aggregation. Also covered: Lakehouse compaction strategies across Delta Lake/Hudi/Iceberg, Spark optimization checklists for production, Gzip pitfalls in Spark on Azure, a Go API stack using Huma/sqlc/Goose, and a Bitcask implementation in the Hare language. ## Content **TLDR:** Jaeger v2.18.0 ships native ClickHouse support as an alpha backend, with benchmark numbers that are hard to ignore. Two BigQuery deep-dives landed today covering what actually happens when you hit Run and why fast queries can still be expensive. Datasette 1.0a30 adds a keyboard-driven jump menu with a plugin hook. Data sketches get a thorough treatment for anyone tired of paying full price for COUNT DISTINCT. --- ## Jaeger v2.18.0 adds ClickHouse as a trace storage backend The Jaeger maintainer who wrote the integration published the benchmark results: 10 million spans across 1 million traces compress from ~6 GiB to ~722 MiB, an 8.6× reduction. Ingestion holds above 50k spans/sec, trace retrieval comes in around 100ms, and most search queries land under 50ms. The schema sorts by `(service_name, name, start_time)` rather than `trace_id` to optimize search, then recovers trace retrieval speed via a bloom filter skip index and a materialized view on trace_id timestamps. Typed span attributes go into ClickHouse Nested columns, and materialized views precompute service names, operations, and trace time ranges so Service Performance Monitoring works without a separate metrics pipeline. ## BigQuery's slot model, explained plainly Two posts today attack BigQuery from different angles and both are worth reading together. The first makes the point that a 5-second query isn't cheap just because it's fast — on-demand billing charges for bytes scanned, not rows returned, and `total_slot_ms` is the number you should actually be watching. The second traces BigQuery's internals back to the 2010 Dremel paper, covering Colossus, the Capacitor columnar format, zone maps, and why partitioning and clustering matter as data-skipping mechanisms rather than performance hints. The honest takeaway from both: most BigQuery problems are architecture and people problems, not SQL problems. ## Datasette 1.0a30 ships a keyboard-driven navigation menu The new "jump to" menu, triggered by `/`, lets you type to navigate directly to a database, table, or canned query. It also exposes a plugin hook so extensions can inject their own entries. Small feature, but the kind of thing that makes a tool feel finished. ## Data sketches: a practical guide to approximate aggregation A thorough writeup covers the mechanics behind probabilistic data structures for streaming aggregation — HyperLogLog, Theta sketches, KLL quantiles, frequent items, and reservoir sampling. The mergeability property is what makes these useful in practice: you can parallelize computation and pre-aggregate without losing the ability to combine results later. The post is honest about the failure modes too: compounding error from stacked set operations, binary format incompatibility between systems, and the hard rule that sketches are wrong for financial or compliance reporting. --- ## Also notable - **Compaction in Lakehouse architectures:** Small files from streaming ingestion, CDC pipelines, and concurrent writers silently kill query planning performance and inflate cloud costs; the post frames the problem across Delta Lake, Hudi, and Iceberg with the deep-dive behind a paywall. - **Apache Spark optimization checklist:** Production-hardened list covering AQE/DPP/SPJ (now default-on in Spark 4.x), skew diagnosis via the Spark UI, splittable compression codecs, and join strategies — each item grounded in real incidents. - **Gzip in Spark pipelines:** A cautionary post on using Gzip with Apache Spark on Azure Blob Storage; Gzip's non-splittable nature caused severe slowdowns, teeing up a codec comparison covering Snappy, LZ4, Zstandard, and Brotli. - **Go API stack without ORMs:** A walkthrough using Huma for HTTP and OpenAPI generation, sqlc for type-safe query generation from plain SQL, and Goose for migrations — keeps SQL explicit and avoids runtime reflection. - **Forgejo on Kubernetes:** Practical notes on deploying a self-hosted Git server with CloudNativePG (3-instance PostgreSQL cluster), PersistentVolumeClaims on Linode, and Traefik via the Gateway API with cert-manager for HTTPS and TCPRoute for SSH. - **Swamp datastores:** The Swamp agent workflow tool now supports pluggable storage backends beyond local SQLite, with a GitLab Terraform state HTTP API backend and a PostgreSQL backend targeting RDS/Aurora with row-level distributed locking and heartbeat TTL for crash recovery. - **Bitcask in Hare:** A developer implements the Bitcask storage engine from scratch in the Hare language for game state persistence, covering binary encoding, a hand-rolled hash map, and plans for key expiry and auto-merge. - **Data consolidation best practices:** A guide covering ETL vs ELT trade-offs, data quality metrics, and tooling comparisons across Fivetran, Talend, Snowflake, and BigQuery — aimed at data engineers building reliable pipelines. ## Similar posts on daily.dev - [How Jaeger hit 8.6× compression on 10 million spans with ClickHouse](https://daily.dev/posts/how-jaeger-hit-8-6-compression-on-10-million-spans-with-clickhouse-n5yas4v4u) · The New Stack · 1 upvotes · 0 comments - [Building Jaeger’s ClickHouse backend: 8.6× compression on 10 million spans](https://daily.dev/posts/building-jaeger-s-clickhouse-backend-8-6-compression-on-10-million-spans-3dmlhztxi) · CNCF · 3 upvotes · 0 comments - [ClickHouse is data lake ready](https://daily.dev/posts/clickhouse-is-data-lake-ready-yi7rw0mlt) · ClickHouse · 2 upvotes · 0 comments --- Tags: [#devops](https://daily.dev/tags/devops), [#apache-spark](https://daily.dev/tags/apache-spark), [#clickhouse](https://daily.dev/tags/clickhouse), [#google-bigquery](https://daily.dev/tags/google-bigquery) [View this post on daily.dev](https://daily.dev/posts/jaeger-hits-8-6-compression-with-clickhouse-bigquery-s-hidden-slot-costs-explained-vta4rjj1g) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"DiscussionForumPosting","mainEntityOfPage":"https://daily.dev/posts/jaeger-hits-8-6-compression-with-clickhouse-bigquery-s-hidden-slot-costs-explained-vta4rjj1g","headline":"Jaeger hits 8.6× compression with ClickHouse, BigQuery's hidden slot costs explained","text":"A data engineering roundup covering several notable developments: Jaeger v2.18.0 ships ClickHouse as an alpha trace storage backend with 8.6× compression and 50k+ spans/sec ingestion; two BigQuery deep-dives explain slot-based billing and columnar internals rooted in the Dremel paper; Datasette 1.0a30 adds a keyboard-driven jump menu with plugin hooks; and a thorough guide covers probabilistic data structures (HyperLogLog, Theta sketches, KLL quantiles) for approximate aggregation. Also covered: Lakehouse compaction strategies across Delta Lake/Hudi/Iceberg, Spark optimization checklists for production, Gzip pitfalls in Spark on Azure, a Go API stack using Huma/sqlc/Goose, and a Bitcask implementation in the Hare language.","url":"https://daily.dev/posts/jaeger-hits-8-6-compression-with-clickhouse-bigquery-s-hidden-slot-costs-explained-vta4rjj1g","datePublished":"2026-05-25T04:21:32.811Z","dateModified":"2026-05-25T04:21:49.944Z","author":{"@type":"Organization","name":"Databases Digest","logo":"https://media.daily.dev/image/upload/s--ir1hdV8b--/f_auto,q_auto/v1773839400/logos/databases_digest?_a=BAMAMiiu0","url":"https://daily.dev/sources/databases_digest"},"interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"isPartOf":{"@type":"WebPage","url":"https://daily.dev/sources/databases_digest","name":"Databases Digest"}} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Databases Digest","item":"https://daily.dev/sources/databases_digest"},{"@type":"ListItem","position":3,"name":"Jaeger hits 8.6× compression with ClickHouse, BigQuery's hidden slot costs explained"}]} ```