--- title: "Can an LLM Forget the Right Things?" url: https://daily.dev/posts/can-an-llm-forget-the-right-things--kvnfq91iw source_url: https://towardsdatascience.com/can-an-llm-forget-the-right-things type: article source: "Towards Data Science" published: 2026-08-24T15:12:26.403Z updated: 2026-08-24T15:12:53.398Z tags: ["deep-learning", "robotics", "ai-inference", "cuda"] reading_time: 24 upvotes: 0 comments: 0 language: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Can an LLM Forget the Right Things? **[Towards Data Science](https://daily.dev/sources/tds)** · 24 min read · 0 upvotes · 0 comments ## Summary A project called vla-edge-backend implements a hand-written CUDA inference runtime designed for robotics vision-language-action workloads, where a hard 33ms control-loop deadline and fixed VRAM budget are treated as first-class constraints. It uses an admission controller (exponential moving average cost estimation) that refuses to start reasoning chunks it can't finish in time, a semantic KV cache eviction policy that evicts the most redundant frame by cosine similarity instead of the oldest, and a lock-free double buffer so a 60Hz camera never blocks on slower reasoning. The entire Qwen2.5-Coder-1.5B-Instruct transformer (RMSNorm, RoPE, grouped-query attention, SwiGLU) is written in raw CUDA with no cuBLAS or libtorch, validated against HuggingFace output at ≥0.999 cosine similarity. The author is explicit this is an architecture demonstration built on a cloud Hopper GPU, not a benchmarked edge deployment, and the transformer itself currently runs about 100x too slow for the deadline it targets. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### How can I make an LLM inference runtime respect a hard real-time deadline instead of just computing until done? Use an admission controller that estimates the cost of the next reasoning chunk with an exponential moving average of prefill and decode times, then refuses to start the chunk if the estimated cost exceeds the remaining deadline budget minus a safety margin. One implementation uses a 33ms deadline with a 2ms safety margin, separate EMAs for prefill (32 tokens) and decode (8 tokens) with alpha 0.2, and falls back to repeating the last action when it declines to admit a chunk. _daily.dev surfaces systems-engineering approaches like this for developers building latency-critical inference pipelines._ ### How should a KV cache decide which frames to evict when memory is full in a vision-language robotics model? Instead of always evicting the oldest frame (FIFO), a semantic eviction policy computes cosine similarity between pooled embeddings of adjacent retained frames and evicts the older frame from whichever adjacent pair is most similar, keeping frames that capture genuinely new information. In one implementation, each frame costs 32 KV cache slots out of 4096 total, giving room for 128 retained frames before eviction kicks in. _developers optimizing memory-constrained inference pipelines can track cache eviction strategies like this on daily.dev._ ### Why does a chat-oriented LLM serving stack like vLLM or TensorRT-LLM break when connected to a live robot camera feed? Three failure modes occur simultaneously: VRAM overflows because chat runtimes assume conversations end, but a camera streams tokens forever; control-loop deadlines get missed silently because standard runtimes have no concept of a hard deadline; and frequency mismatch occurs because a 60Hz camera outpaces the reasoning model, forcing perception to either block or accumulate a stale backlog unless something explicitly decouples the two. _engineers evaluating inference stacks for real-time robotics workloads can follow this kind of architecture breakdown on daily.dev._ ## Similar posts on daily.dev - [Build Next-Gen Physical AI with Edge‑First LLMs for Autonomous Vehicles and Robotics](https://daily.dev/posts/build-next-gen-physical-ai-with-edge-first-llms-for-autonomous-vehicles-and-robotics-gqwjnidwm) · NVIDIA Developer · 1 upvotes · 0 comments - [Accelerating LLM and VLM Inference for Automotive and Robotics with NVIDIA TensorRT Edge-LLM](https://daily.dev/posts/accelerating-llm-and-vlm-inference-for-automotive-and-robotics-with-nvidia-tensorrt-edge-llm-1lrxf6i15) · NVIDIA Developer · 1 upvotes · 0 comments - [xaskasdf/ntransformer: High-efficiency LLM inference engine in C\+\+/CUDA. Run Llama 70B on RTX 3090.](https://daily.dev/posts/xaskasdf-ntransformer-high-efficiency-llm-inference-engine-in-c-cuda-run-llama-70b-on-rtx-3090--ahx4zgqks) · Hacker News · 1 upvotes · 0 comments --- Tags: [#deep-learning](https://daily.dev/tags/deep-learning), [#robotics](https://daily.dev/tags/robotics), [#ai-inference](https://daily.dev/tags/ai-inference), [#cuda](https://daily.dev/tags/cuda) [View this post on daily.dev](https://daily.dev/posts/can-an-llm-forget-the-right-things--kvnfq91iw)