--- title: Why real-time AI at scale is so hard | daily.dev description: Real-time AI systems often fail in production due to infrastructure issues rather than model problems. Common failure modes include P99 tail latency spikes... canonical: https://daily.dev/posts/why-real-time-ai-at-scale-is-so-hard-5xrkdsob1 twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Why real-time AI at scale is so hard | daily.dev og:description: Real-time AI systems often fail in production due to infrastructure issues rather than model problems. Common failure modes include P99 tail latency spikes... og:url: https://daily.dev/posts/why-real-time-ai-at-scale-is-so-hard-5xrkdsob1 og:image: https://api.daily.dev/og/posts/5xrkDSob1.png og:image:alt: Why real-time AI at scale is so hard og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Why real-time AI at scale is so hard **[The New Stack](https://daily.dev/sources/newstack)** · 9 min read · 1 upvotes · 0 comments ## Summary Real-time AI systems often fail in production due to infrastructure issues rather than model problems. Common failure modes include P99 tail latency spikes from write contention, stale features degrading accuracy despite good offline metrics, vector index rot from repeated re-embedding (recall dropping to 42% in one case), resource contention between training and serving, and disruption during model retraining/cutover. The piece, sponsored by ScyllaDB, argues these are classic distributed systems problems and recommends monitoring freshness/backlog/index health, isolating workloads, separating vector indexing from the core database, and using storage engines that absorb write bursts (citing ScyllaDB's lock-free write path and tablet-based elastic scaling as examples, with a billion-vector benchmark showing sub-10ms P99 latency at 300 concurrency). ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### Why does P99 latency spike suddenly in a real-time AI feature-serving system even when average latency looks fine? Tail latency spikes because of lock contention under highly concurrent write throughput, not because of a slow model. In one financial trading application, P99 latency jumped to 3 seconds at around 740,000 operations per second because feature lookup reads queued up behind writes; retries, bigger caches, or connection pool tuning did not fix it since tail latency is a property of the underlying architecture. _Anyone debugging tail latency in an ML pipeline can compare architecture tradeoffs like these on daily.dev._ ### Why would offline evaluation metrics for a machine learning model look great but production accuracy be poor? Stale features are usually the cause: online serving data can drift far past freshness targets (in one case, user profile staleness blew past a five-minute SLA by hours) while offline evaluation uses clean, current data. This mismatch means the model makes decisions on outdated inputs, producing a garbage-in, garbage-out accuracy drop that has nothing to do with the model itself. _Teams chasing accuracy drops can track feature-freshness pitfalls like this on daily.dev._ ### Why does vector search recall degrade over time in an HNSW index without any code changes? HNSW graphs degrade as they accumulate mutations from repeated re-embedding passes, causing recall and query latency to worsen even though nothing appears broken. In one reported case, recall dropped to 42% after repeated re-embedding. Changing the similarity function, search parameters, or embedding model requires rebuilding the graph from scratch, so vector indexes need the same ongoing maintenance as any database index. _Developers maintaining vector search pipelines can follow index-health tradeoffs like these on daily.dev._ ## Similar posts on daily.dev - [The 200ms latency: A developer’s guide to real-time personalization](https://daily.dev/posts/the-200ms-latency-a-developer-s-guide-to-real-time-personalization-u7s5bwkwz) · InfoWorld · 0 upvotes · 0 comments - [Why the secret to scaling AI isn’t a better model, it’s a simpler foundation](https://daily.dev/posts/why-the-secret-to-scaling-ai-isn-t-a-better-model-it-s-a-simpler-foundation-xepzjzznp) · The New Stack · 1 upvotes · 0 comments - [Your Pipeline Is Real-Time. Your AI Is Still Ten Minutes Behind.](https://daily.dev/posts/your-pipeline-is-real-time-your-ai-is-still-ten-minutes-behind--wlsw7755c) · Medium · 1 upvotes · 0 comments - [Agentic AI: Why Enterprise AI Is Not Delivering on Its Promise](https://daily.dev/posts/agentic-ai-why-enterprise-ai-is-not-delivering-on-its-promise-szxzvstfv) · SingleStore · 2 upvotes · 0 comments - [Stop Treating Your Models Like Microservices](https://daily.dev/posts/stop-treating-your-models-like-microservices-pwqhglvbb) · Cloud Native Now · 0 upvotes · 0 comments --- Tags: [#postgresql](https://daily.dev/tags/postgresql), [#vector-search](https://daily.dev/tags/vector-search), [#scylladb](https://daily.dev/tags/scylladb) [View this post on daily.dev](https://daily.dev/posts/why-real-time-ai-at-scale-is-so-hard-5xrkdsob1) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Why real-time AI at scale is so hard","url":"https://daily.dev/posts/why-real-time-ai-at-scale-is-so-hard-5xrkdsob1","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/why-real-time-ai-at-scale-is-so-hard-5xrkdsob1"},"datePublished":"2026-08-23T16:04:23.119Z","dateModified":"2026-09-13T18:33:09.103Z","description":"Real-time AI systems often fail in production due to infrastructure issues rather than model problems. Common failure modes include P99 tail latency spikes...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/589d8dc99547739b58f1fe48260792f8?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/589d8dc99547739b58f1fe48260792f8?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"The New Stack","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"The New Stack","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/newstack","url":"https://daily.dev/sources/newstack"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/why-real-time-ai-at-scale-is-so-hard-5xrkdsob1","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":1},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"postgresql,vector-search,scylladb","timeRequired":"PT9M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"The New Stack","item":"https://daily.dev/sources/newstack"},{"@type":"ListItem","position":3,"name":"Why real-time AI at scale is so hard"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/why-real-time-ai-at-scale-is-so-hard-5xrkdsob1#faq","mainEntity":[{"@type":"Question","name":"Why does P99 latency spike suddenly in a real-time AI feature-serving system even when average latency looks fine?","acceptedAnswer":{"@type":"Answer","text":"Tail latency spikes because of lock contention under highly concurrent write throughput, not because of a slow model. In one financial trading application, P99 latency jumped to 3 seconds at around 740,000 operations per second because feature lookup reads queued up behind writes; retries, bigger caches, or connection pool tuning did not fix it since tail latency is a property of the underlying architecture. Anyone debugging tail latency in an ML pipeline can compare architecture tradeoffs like these on daily.dev."}},{"@type":"Question","name":"Why would offline evaluation metrics for a machine learning model look great but production accuracy be poor?","acceptedAnswer":{"@type":"Answer","text":"Stale features are usually the cause: online serving data can drift far past freshness targets (in one case, user profile staleness blew past a five-minute SLA by hours) while offline evaluation uses clean, current data. This mismatch means the model makes decisions on outdated inputs, producing a garbage-in, garbage-out accuracy drop that has nothing to do with the model itself. Teams chasing accuracy drops can track feature-freshness pitfalls like this on daily.dev."}},{"@type":"Question","name":"Why does vector search recall degrade over time in an HNSW index without any code changes?","acceptedAnswer":{"@type":"Answer","text":"HNSW graphs degrade as they accumulate mutations from repeated re-embedding passes, causing recall and query latency to worsen even though nothing appears broken. In one reported case, recall dropped to 42% after repeated re-embedding. Changing the similarity function, search parameters, or embedding model requires rebuilding the graph from scratch, so vector indexes need the same ongoing maintenance as any database index. Developers maintaining vector search pipelines can follow index-health tradeoffs like these on daily.dev."}}]} ```