A newsletter-style digest covers three separate items. First, LMCache, an open-source library, moves KV cache management out of the inference engine's process into a separate one, avoiding GPU idle time while cache blocks move between memory tiers; on H200s running Qwen3-235B it delivers 14x faster time-to-first-token and 4x faster decoding. Second, a study (sponsored content tied to Sonar's Vortex product) argues that most AI coding agent token spend goes toward locating code via text search rather than writing it, since text search struggles with ambiguous names, overloaded identifiers, and structural relationships not expressed in shared wording; treating the codebase as a graph of nodes and edges lets agents ask direct structural questions, cutting cost by 5-36% across six tasks in four languages in a controlled test. Third, a short explainer on label smoothing as a regularization technique, showing improved generalization on Fashion MNIST but reduced model confidence.
Table of contents
Your KV cache library is stealing your throughputHow semantic code navigation cuts agent token costs by up to 36%Label smoothing for regularizationQuestions this post answers
Why do AI coding agent token bills get so expensive even when the agent writes relatively little code?
Most of the token spend goes toward locating the right place to change code, not writing it. Agents typically rely only on text search to build a map of a codebase, which fails when a name appears in hundreds of irrelevant places, when two things share a name but differ in meaning, or when code is connected structurally (like an interface implementation) without shared wording, forcing the agent to read and reason over far more content than necessary. Anyone tracking runaway AI coding costs can follow token-efficiency techniques like this on daily.dev.
How much can semantic code navigation reduce AI coding agent costs compared to plain text search?
In a controlled test replaying real, previously merged open-source commits, cost fell in every one of six tasks across four languages when agents used a graph-based semantic navigation approach instead of plain text search, ranging from a 5% reduction on a straightforward change up to 36% on a Java interface change. Each task ran ten times per approach and had to pass the actual build and tests to count. Developers comparing agent tooling approaches can dig into benchmarks like this on daily.dev.
Why does putting KV cache management in a separate process from the inference engine improve throughput?
Cache management and attention computation have opposite bottlenecks: moving cached tensors between GPU, CPU, and disk waits on I/O, while running attention is GPU-bound math. When both run in one process, the engine handles them sequentially, leaving the GPU idle during I/O waits. LMCache separates cache management into its own process sharing GPU memory with the engine, passing only block IDs, which stopped cache work from stalling inference and delivered 14x faster time-to-first-token on H200s running Qwen3-235B. Engineers tuning inference stacks for throughput can track approaches like this on daily.dev.