<p><strong>TLDR:</strong> Discord’s voice service collapsed for three hours after a Kubernetes grace period mismatch triggered millions of simultaneous reconnections, cascading into a full cluster failure. DuckDB announced Quack, a native HTTP-based remote protocol claiming 3.5x faster transfers than Arrow Flight, available now in v1.5.3. Stripe’s selective test execution system runs only 5% of tests on average across a 50-million-line Ruby monorepo by intercepting syscalls at runtime. LinkedIn engineers used eBPF off-CPU profiling to diagnose ephemeral 10-15 second database outages with no obvious external trigger.</p>
<hr>
<h2>Discord voice outage: Kubernetes grace period mismatch caused thundering herd, took down 14 of 15 voice instances</h2>
<p>Discord’s voice service dropped 17% of global sessions after engineers tried to vertically scale the session service. A safety check ran longer than the Kubernetes termination grace period, so pods were marked dead mid-check and triggered millions of simultaneous reconnection attempts. The surge exhausted memory in US East, cascaded globally, and overwhelmed the Erlang-based voice connection service — 14 of 15 instances went down. Both targeted and full cluster restarts failed under the connection flood. Recovery required aggressive rate limiting plus doubling cluster size, with the outage lasting roughly three hours. The takeaway is blunt: rate limiting and back-pressure are not optional in distributed systems, and termination grace periods must be longer than any safety check that runs during shutdown. <a href="https://app.daily.dev/posts/AU2roeX5G" target="_blank" rel="noopener nofollow">Read more</a></p>
<h2>DuckDB Quack: native HTTP protocol enables multi-user analytics, claims 3.5x faster than Arrow Flight</h2>
<p>DuckDB has shipped Quack, an HTTP-based remote protocol that lets multiple DuckDB instances share the same database over a network — something previously impossible with DuckDB’s embedded-only model. The team claims Quack transfers large datasets roughly 3.5x faster than Arrow Flight and significantly faster than PostgreSQL, and deliberately avoided adopting Arrow Flight SQL to retain full control over protocol evolution. Quack is available as an autoloadable core extension in DuckDB v1.5.3, with DuckLake integration, replication, and a production-ready release targeting DuckDB 2.0 later in 2026. If the performance numbers hold up under real workloads, this changes the calculus for teams using DuckDB in shared analytics environments. <a href="https://app.daily.dev/posts/kcLxReZZV" target="_blank" rel="noopener nofollow">Read more</a></p>
<h2>Stripe’s selective test execution runs 5% of tests on a 50M-line Ruby monorepo using runtime syscall interception</h2>
<p>Stripe built a C++ shared library loaded via LD_PRELOAD that intercepts open syscalls and records which files each test actually accesses at runtime, building a file-level dependency graph stored as roaring bitmaps. Static analysis was a dead end given Ruby’s dynamic nature. When a build starts, changed files are detected via hashdeep, mapped to bitmap indexes, and unioned to find impacted tests — the median run executes 0.5% of the 1.2 million test units, using less than 10% of full-run compute. The system includes guardrails for directory-globbing tests, previously failing tests, and linters. Build baseline metadata lives in MongoDB with Monotonic Revision IDs for reproducible selection. <a href="https://app.daily.dev/posts/xqE4QWaTP" target="_blank" rel="noopener nofollow">Read more</a></p>
<h2>LinkedIn used eBPF off-CPU profiling to find the root cause of ephemeral 10-15 second database outages</h2>
<p>LinkedIn’s user feed database was experiencing recurring 10-15 second outages that were ephemeral, patternless, and had no clear external trigger — the kind of failure that defeats traditional CPU profiling because the process isn’t doing anything visible when it’s stuck. The team turned to off-CPU profiling with eBPF, which captures what a process is waiting on rather than what it’s actively executing. This is a useful technique to have in your toolkit for diagnosing latency spikes caused by lock contention, I/O waits, or scheduler delays that don’t show up in CPU flame graphs. <a href="https://app.daily.dev/posts/72Gpul57S" target="_blank" rel="noopener nofollow">Read more</a></p>
<hr>
<h2>Also notable</h2>
<ul>
<li><strong>HTTP retry client in Go: request body rewinding, jitter, Retry-After header handling, and thundering herd warnings:</strong> A production-grade walkthrough covers the non-obvious parts of building a Go HTTP retry client: rewinding request bodies between attempts, exponential backoff with full jitter to prevent thundering herds, context-aware cancellable sleep, skipping non-idempotent methods like POST by default, draining response bodies for connection reuse, and handling both delta-seconds and HTTP-date forms of Retry-After — with a recommendation to use go-retryablehttp or resty in production rather than rolling your own. <a href="https://app.daily.dev/posts/ODcHd7f4T" target="_blank" rel="noopener nofollow">Read more</a></li>
<li><strong>PGlite runs full PostgreSQL in-process via WASM, enabling integration tests in under 3 seconds with no Docker:</strong> @electric-sql/pglite compiles PostgreSQL to WebAssembly and runs it entirely in-process, supporting real SQL, JSONB, array types, foreign keys, and gen_random_uuid() — the full test suite for a tRPC router completes in under 3 seconds with no external infrastructure required. <a href="https://app.daily.dev/posts/JEMyrX7sy" target="_blank" rel="noopener nofollow">Read more</a></li>
<li><strong>Transactional outbox pattern vs RabbitMQ DLQ: which to use when writing to MongoDB and ClickHouse:</strong> Writing to two databases creates a silent consistency gap if the app crashes between writes; the outbox pattern eliminates this by atomically writing both the entity and an outbox record in the same MongoDB transaction, then using a relay worker with at-least-once semantics and ReplacingMergeTree deduplication for ClickHouse — RabbitMQ with DLQ is reserved for sub-second latency or multiple-consumer requirements. <a href="https://app.daily.dev/feed-by-ids?id=1vBubWl2g&amp;id=U7zUXN4Cu" target="_blank" rel="noopener nofollow">Read more</a></li>
<li><strong>Phi Accrual failure detection produces a continuous suspicion value instead of binary up/down, enabling graduated responses:</strong> Unlike heartbeat-based detectors, Phi Accrual computes φ = -log10(probability of receiving a heartbeat) from a sliding window of past arrival times, letting you define multiple thresholds that trigger progressively more drastic responses rather than a single hard cutover — useful for distributed systems where you want to degrade gracefully before declaring a node dead. <a href="https://app.daily.dev/posts/YY3BH7Gav" target="_blank" rel="noopener nofollow">Read more</a></li>
<li><strong>NTP packets are unauthenticated; ARP spoofing plus forged NTP can shift system clocks and break TOTP, TLS, and distributed data:</strong> Linux’s systemd-timesyncd accepts any time offset from a forged NTP packet, enabling disruption of timed events, TOTP windows, TLS certificate validity, and distributed system ordering — Network Time Security (NTS) adds authenticated key exchange but adoption remains limited. <a href="https://app.daily.dev/posts/TwmjGJAJK" target="_blank" rel="noopener nofollow">Read more</a></li>
</ul>


Server-side development, system design, API patterns, and distributed architecture.

Backend Digest

A curated roundup of notable engineering stories: Discord's voice service suffered a three-hour outage after a Kubernetes grace period mismatch triggered millions of simultaneous reconnections, taking down 14 of 15 voice instances. DuckDB shipped Quack, a native HTTP protocol enabling multi-user analytics claiming 3.5x faster transfers than Arrow Flight, available in v1.5.3. Stripe's selective test execution system uses runtime syscall interception to run only 5% of tests on a 50-million-line Ruby monorepo. LinkedIn used eBPF off-CPU profiling to diagnose ephemeral 10-15 second database outages. Also covered: Go HTTP retry client patterns, PGlite running PostgreSQL in-process via WASM, transactional outbox vs RabbitMQ DLQ, Phi Accrual failure detection, and NTP authentication vulnerabilities.

Discord's thundering herd takes down voice, DuckDB Quack ships multi-user HTTP protocol