--- title: Runbooks and rollbacks keep lying, while agent infra... description: Today's theme is decay you can't see until it breaks: runbooks that render fine but point at dead infrastructure, feature flags that survive a rollback and... canonical: https://daily.dev/posts/runbooks-and-rollbacks-keep-lying-while-agent-infra-becomes-the-new-battleground-qxvepewdr twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Runbooks and rollbacks keep lying, while agent infra becomes the new battleground | daily.dev og:description: Today's theme is decay you can't see until it breaks: runbooks that render fine but point at dead infrastructure, feature flags that survive a rollback and... og:url: https://daily.dev/posts/runbooks-and-rollbacks-keep-lying-while-agent-infra-becomes-the-new-battleground-qxvepewdr og:image: https://api.daily.dev/og/posts/QXvEpewDR.png og:image:alt: Runbooks and rollbacks keep lying, while agent infra becomes the new battleground og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Runbooks and rollbacks keep lying, while agent infra becomes the new battleground **[Backend Digest](https://daily.dev/sources/backend_digest)** · 5 min read · 0 upvotes · 0 comments ## Summary Today's theme is decay you can't see until it breaks: runbooks that render fine but point at dead infrastructure, feature flags that survive a rollback and keep routing into broken code, and database-per-service architectures that never assigned an owner to the transaction between services. Buildkite's postmortem shows how interacting Kubernetes control loops turned a routine deploy into a full outage. On the agent infrastructure side, Vercel disclosed serving a trillion requests a month through its unified Fluid compute layer, and Databricks found seven small MCP tool bugs were quietly burning over a million dollars a year in wasted agent spend. ## Content **TLDR:** Today's theme is decay you can't see until it breaks: runbooks that render fine but point at dead infrastructure, feature flags that survive a rollback and keep routing into broken code, and database-per-service architectures that never assigned an owner to the transaction between services. Buildkite's postmortem shows how interacting Kubernetes control loops turned a routine deploy into a full outage. On the agent infrastructure side, Vercel disclosed serving a trillion requests a month through its unified Fluid compute layer, and Databricks found seven small MCP tool bugs were quietly burning over a million dollars a year in wasted agent spend. --- ## Three postmortems, one lesson: your recovery plan has silent assumptions A runbook failed at step 17 because it told responders to drain traffic through a load balancer decommissioned five months earlier - the document still rendered fine, it just pointed at nothing. Separately, a Kubernetes rollback restored old code in four minutes, but the feature flag stayed on and kept routing customers into a workflow the old binary couldn't handle. And a database-per-service shop found nobody owned the transaction between billing, identity, and storage during an account-closure workflow, until they built a durable workflow record to own coordination, retries, and reconciliation explicitly. Same root cause across all three: deployments, flags, schema, and docs are independently changing layers, and nothing forces you to check whether last month's architecture change invalidated this week's recovery plan. [Read more](https://daily.dev/feed-by-ids?id=YcGKP1tID&id=vl7ONZtMu&id=ttvg8cIsZ) ## Buildkite's autoscaler feedback loop cascaded into a full outage Buildkite's August 25 incident started with a deploy eating all cluster headroom while background workers tried to scale out simultaneously, delaying autoscaling and triggering runaway replica requests under an unusually high maxReplicas setting. When new nodes finally came online, the surge of new pods and endpoints overwhelmed CoreDNS, which wasn't configured to autoscale, and DNS failure cascaded into API, job dispatch, and notification failures for every customer. Recovery required manually pausing deploys and scaling CoreDNS directly - a good reminder that HPA and cluster autoscaler are two control loops that can fight each other in ways neither team notices until it's too late. [Read more](https://daily.dev/posts/czbo1IYGO) ## Vercel's Fluid compute now handles a trillion requests a month Vercel detailed Fluid, the unified compute architecture behind builds, functions, and sandboxes, built from three pieces: Hive (VM provisioning at global scale), a custom boot format called VHS that boots arbitrary OS images in milliseconds, and Vercel Drives (portable storage decoupled from compute, still in private beta). The numbers are the story here: 15 million builds a day, 25 million sandboxes a week, a trillion requests a month. It's explicitly positioned around the fast-provisioning needs of agent workloads, not just traditional web traffic. [Read more](https://daily.dev/posts/8Fxq1LMNR) ## Seven silent MCP bugs were costing Databricks $1.2M a year Databricks engineers found seven small bugs in their MCP tool servers were burning $499K/year in wasted tokens plus 12,000 engineering hours annually, because agents were quietly retrying failed tool calls instead of surfacing errors. Using OpenTelemetry tracing on MCP calls plus natural-language querying of the trace table, they found and fixed all seven bugs in one hour. The root cause was under-specified tool signatures - models made reasonable guesses (passing a JSON array instead of a comma-separated string) that crashed the tool instead of degrading gracefully. If you're building tools for agents, this is the case study for why error message quality is now a cost lever, not just a UX nicety. [Read more](https://daily.dev/posts/sl4y8p7Za) --- ## Also notable - **Debezium 3.6.2.Final patches a PostgreSQL JDBC CVE and a cache leak:** The release fixes gradual throughput degradation from an unbounded internal cache, adds incremental snapshot support on PostgreSQL read replicas, and upgrades the PostgreSQL JDBC driver to 42.7.13 to address CVE-2026-54291. [Read more](https://daily.dev/posts/zcESADH7x) - **Fastify's year of npm hardening pins every GitHub Action to a commit SHA:** Prompted by GitHub's OSS Fund, Fastify maintainers removed risky pull_request_target workflows, pinned all Actions to commit SHAs, and adopted npm's min-release-age=2 setting specifically to blunt lifecycle-script attacks like the September 2025 Shai-Hulud campaign. [Read more](https://daily.dev/posts/y51RgOVbL) - **Uken Games cut observability spend 87% by replacing Datadog with ClickHouse:** The mobile studio's self-hosted stack (ClickHouse, OpenTelemetry, SigNoz, Prometheus, Grafana) dropped annual costs from six figures to about $12,000 a year while storing all traces on a single ClickHouse node using roughly 170GB of disk. [Read more](https://daily.dev/posts/nDgLfU6JO) - **ClickHouse's agent benchmark finds the cheapest model wins on real analytics tasks:** Testing 28 models on 201 real analytics questions, Claude Fable 5.1 led correctness at 76.6%, but DeepSeek V4 Flash hit 65.7% at 52x lower cost ($1 vs $52 for the full run), with 'wrong plan' rather than SQL syntax as the dominant failure mode. [Read more](https://daily.dev/posts/81A3CONc5) - **pnpm 12's Rust rewrite cuts install times up to 90% with zero migration:** Commands, flags, and lockfile format are unchanged from pnpm 11, but Vercel's Turborepo benchmarks showed median install times dropping 64-90% versus pnpm 10.28, with peer dependency resolution 2-3x faster. [Read more](https://daily.dev/posts/km4JaECaK) ## Similar posts on daily.dev - [How we eliminated $1 million a year of wasted AI agent spend in one hour](https://daily.dev/posts/how-we-eliminated-1-million-a-year-of-wasted-ai-agent-spend-in-one-hour-sl4y8p7za) · databricks · 1 upvotes · 0 comments --- Tags: [#kubernetes](https://daily.dev/tags/kubernetes), [#observability](https://daily.dev/tags/observability), [#mcp](https://daily.dev/tags/mcp) [View this post on daily.dev](https://daily.dev/posts/runbooks-and-rollbacks-keep-lying-while-agent-infra-becomes-the-new-battleground-qxvepewdr) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"DiscussionForumPosting","mainEntityOfPage":"https://daily.dev/posts/runbooks-and-rollbacks-keep-lying-while-agent-infra-becomes-the-new-battleground-qxvepewdr","headline":"Runbooks and rollbacks keep lying, while agent infra becomes the new battleground","text":"Today's theme is decay you can't see until it breaks: runbooks that render fine but point at dead infrastructure, feature flags that survive a rollback and keep routing into broken code, and database-per-service architectures that never assigned an owner to the transaction between services. Buildkite's postmortem shows how interacting Kubernetes control loops turned a routine deploy into a full outage. On the agent infrastructure side, Vercel disclosed serving a trillion requests a month through its unified Fluid compute layer, and Databricks found seven small MCP tool bugs were quietly burning over a million dollars a year in wasted agent spend.","url":"https://daily.dev/posts/runbooks-and-rollbacks-keep-lying-while-agent-infra-becomes-the-new-battleground-qxvepewdr","datePublished":"2026-09-02T04:18:05.833Z","dateModified":"2026-09-02T04:18:28.031Z","author":{"@type":"Organization","name":"Backend Digest","logo":"https://media.daily.dev/image/upload/s--9uCSrsAb--/f_auto,q_auto/v1773839398/logos/backend_digest?_a=BAMAMiiu0","url":"https://daily.dev/sources/backend_digest"},"interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"isPartOf":{"@type":"WebPage","url":"https://daily.dev/sources/backend_digest","name":"Backend Digest"}} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Backend Digest","item":"https://daily.dev/sources/backend_digest"},{"@type":"ListItem","position":3,"name":"Runbooks and rollbacks keep lying, while agent infra becomes the new battleground"}]} ```