<!-- mobian-agent-page publisher="dailydev" canonical="https://daily.dev/posts/vertical-mobility-inference-from-mvp-to-trillion-parameter-workloads-sitanshu-gupta-coreweave-wrjpdvys8" -->

---
title: Vertical Mobility: Inference from MVP to...
description: A CoreWeave inference lead describes the architecture behind their multi-tenant inference platform, covering serverless (pay-per-token, provisioned throughput)...
canonical: https://daily.dev/posts/vertical-mobility-inference-from-mvp-to-trillion-parameter-workloads-sitanshu-gupta-coreweave-wrjpdvys8
twitter:card: summary_large_image
twitter:site: @dailydotdev
og:type: website
og:site_name: daily.dev
og:title: Vertical Mobility: Inference from MVP to Trillion-Parameter Workloads — Sitanshu Gupta, CoreWeave | daily.dev
og:description: A CoreWeave inference lead describes the architecture behind their multi-tenant inference platform, covering serverless (pay-per-token, provisioned throughput)...
og:url: https://daily.dev/posts/vertical-mobility-inference-from-mvp-to-trillion-parameter-workloads-sitanshu-gupta-coreweave-wrjpdvys8
og:image: https://api.daily.dev/og/posts/WRJpDvyS8.png
og:image:alt: Vertical Mobility: Inference from MVP to Trillion-Parameter Workloads — Sitanshu Gupta, CoreWeave
og:image:width: 1200
og:image:height: 630
og:locale: en
---

> ## Documentation Index
> Fetch the complete documentation index at: https://daily.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# Vertical Mobility: Inference from MVP to Trillion-Parameter Workloads — Sitanshu Gupta, CoreWeave

**[AI Engineer](https://daily.dev/sources/aidotengineer)** · 15 min read · 0 upvotes · 0 comments

## Summary

A CoreWeave inference lead describes the architecture behind their multi-tenant inference platform, covering serverless (pay-per-token, provisioned throughput) and dedicated (per-GPU-hour) consumption models. The talk details workload shapes (agentic, chat, real-time voice/video, batch), KV-cache-aware routing for prefill reuse, prefill-decode disaggregation choices, offloading KV cache to high-bandwidth storage using techniques like LMCache and Mooncake, and performance levers such as NVFP4 quantization and speculative decoding trained per customer. It closes with benchmark comparisons against Fireworks on artificial analysis and OpenRouter leaderboards.

## Full article

daily.dev links to this article rather than hosting it. Read it at the original source: <https://www.youtube.com/watch?v=cQQbJqvZkpo>

## Questions this post answers

### What is prefill-decode disaggregation in LLM inference and when should it be used?

Prefill-decode disaggregation separates the compute-bound prefill phase from the decode phase onto different hardware resources, which can improve efficiency for certain workloads. It is not cheap for every use case, so platforms should provide it as an optional capability rather than a default, letting customers choose based on their specific workload characteristics and cost tolerance.

_Engineers weighing inference architecture trade-offs can follow platform design deep dives like this on daily.dev._

### How does KV cache reuse reduce cost in agentic LLM workloads?

Agentic workloads typically have very high input sequence lengths, with 80-90% of that input often repeated across requests, so reusing cached KV values avoids recomputing the expensive prefill step. Since prefill is compute-bound and costly, hitting the cache more often significantly lowers cost, which is why cached input tokens are priced much cheaper than fresh input tokens.

_daily.dev helps engineers optimizing LLM serving costs keep up with caching techniques like this._

### What techniques exist for offloading KV cache to reduce latency in multi-turn chat applications?

KV cache can be offloaded to high-bandwidth storage between conversation turns instead of being evicted, allowing it to be reloaded quickly when a user returns to the same chat. Open techniques for this include LMCache and Mooncake, which store prefill computations so subsequent requests in the same conversation avoid redoing expensive prefill work.

_Teams building latency-sensitive chat systems can track KV cache offloading approaches on daily.dev._

## Similar posts on daily.dev

- [The LLM Inference Trilemma: Throughput, Latency, Cost](https://daily.dev/posts/the-llm-inference-trilemma-throughput-latency-cost-ws3duzsii) · DigitalOcean · 0 upvotes · 0 comments
- [GPU and TPU utilization with multi-cluster GKE Inference Gateway](https://daily.dev/posts/gpu-and-tpu-utilization-with-multi-cluster-gke-inference-gateway-zu3hs1yh0) · Google Cloud · 0 upvotes · 0 comments
- [Chip Huyen explains how to cut inference costs without new hardware](https://daily.dev/posts/chip-huyen-explains-how-to-cut-inference-costs-without-new-hardware-tjigfbzjs) · The New Stack · 0 upvotes · 0 comments
- [Reliable LLM Inference at Scale](https://daily.dev/posts/reliable-llm-inference-at-scale-dx5jtkvej) · databricks · 0 upvotes · 0 comments

---

Tags: [#gpu](https://daily.dev/tags/gpu), [#ai-inference](https://daily.dev/tags/ai-inference), [#vllm](https://daily.dev/tags/vllm)

[View this post on daily.dev](https://daily.dev/posts/vertical-mobility-inference-from-mvp-to-trillion-parameter-workloads-sitanshu-gupta-coreweave-wrjpdvys8)

```json
{"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]}
{"@context":"https://schema.org","@type":"TechArticle","headline":"Vertical Mobility: Inference from MVP to Trillion-Parameter Workloads — Sitanshu Gupta, CoreWeave","url":"https://daily.dev/posts/vertical-mobility-inference-from-mvp-to-trillion-parameter-workloads-sitanshu-gupta-coreweave-wrjpdvys8","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/vertical-mobility-inference-from-mvp-to-trillion-parameter-workloads-sitanshu-gupta-coreweave-wrjpdvys8"},"datePublished":"2026-09-19T16:40:17.916Z","dateModified":"2026-09-19T16:40:42.525Z","description":"A CoreWeave inference lead describes the architecture behind their multi-tenant inference platform, covering serverless (pay-per-token, provisioned throughput)...","image":"https://i.ytimg.com/vi/cQQbJqvZkpo/sddefault.jpg","thumbnailUrl":"https://i.ytimg.com/vi/cQQbJqvZkpo/sddefault.jpg","isAccessibleForFree":true,"articleSection":"AI Engineer","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"AI Engineer","logo":"https://media.daily.dev/image/upload/s--u5PucxNT--/f_auto/v1724338940/logos/aidotengineer","url":"https://daily.dev/sources/aidotengineer"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/vertical-mobility-inference-from-mvp-to-trillion-parameter-workloads-sitanshu-gupta-coreweave-wrjpdvys8","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"gpu,ai-inference,vllm","timeRequired":"PT15M","video":{"@type":"VideoObject","name":"Vertical Mobility: Inference from MVP to Trillion-Parameter Workloads — Sitanshu Gupta, CoreWeave","description":"A CoreWeave inference lead describes the architecture behind their multi-tenant inference platform, covering serverless (pay-per-token, provisioned throughput)...","thumbnailUrl":"https://i.ytimg.com/vi/cQQbJqvZkpo/sddefault.jpg","uploadDate":"2026-09-19T16:40:17.916Z","duration":"PT15M","url":"https://api.daily.dev/r/WRJpDvyS8","embedUrl":"https://www.youtube.com/embed/cQQbJqvZkpo"}}
{"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"AI Engineer","item":"https://daily.dev/sources/aidotengineer"},{"@type":"ListItem","position":3,"name":"Vertical Mobility: Inference from MVP to Trillion-Parameter Workloads — Sitanshu Gupta, CoreWeave"}]}
{"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/vertical-mobility-inference-from-mvp-to-trillion-parameter-workloads-sitanshu-gupta-coreweave-wrjpdvys8#faq","mainEntity":[{"@type":"Question","name":"What is prefill-decode disaggregation in LLM inference and when should it be used?","acceptedAnswer":{"@type":"Answer","text":"Prefill-decode disaggregation separates the compute-bound prefill phase from the decode phase onto different hardware resources, which can improve efficiency for certain workloads. It is not cheap for every use case, so platforms should provide it as an optional capability rather than a default, letting customers choose based on their specific workload characteristics and cost tolerance. Engineers weighing inference architecture trade-offs can follow platform design deep dives like this on daily.dev."}},{"@type":"Question","name":"How does KV cache reuse reduce cost in agentic LLM workloads?","acceptedAnswer":{"@type":"Answer","text":"Agentic workloads typically have very high input sequence lengths, with 80-90% of that input often repeated across requests, so reusing cached KV values avoids recomputing the expensive prefill step. Since prefill is compute-bound and costly, hitting the cache more often significantly lowers cost, which is why cached input tokens are priced much cheaper than fresh input tokens. daily.dev helps engineers optimizing LLM serving costs keep up with caching techniques like this."}},{"@type":"Question","name":"What techniques exist for offloading KV cache to reduce latency in multi-turn chat applications?","acceptedAnswer":{"@type":"Answer","text":"KV cache can be offloaded to high-bandwidth storage between conversation turns instead of being evicted, allowing it to be reloaded quickly when a user returns to the same chat. Open techniques for this include LMCache and Mooncake, which store prefill computations so subsequent requests in the same conversation avoid redoing expensive prefill work. Teams building latency-sensitive chat systems can track KV cache offloading approaches on daily.dev."}}]}
```

