A project called vla-edge-backend implements a hand-written CUDA inference runtime designed for robotics vision-language-action workloads, where a hard 33ms control-loop deadline and fixed VRAM budget are treated as first-class constraints. It uses an admission controller (exponential moving average cost estimation) that refuses to start reasoning chunks it can't finish in time, a semantic KV cache eviction policy that evicts the most redundant frame by cosine similarity instead of the oldest, and a lock-free double buffer so a 60Hz camera never blocks on slower reasoning. The entire Qwen2.5-Coder-1.5B-Instruct transformer (RMSNorm, RoPE, grouped-query attention, SwiGLU) is written in raw CUDA with no cuBLAS or libtorch, validated against HuggingFace output at ≥0.999 cosine similarity. The author is explicit this is an architecture demonstration built on a cloud Hopper GPU, not a benchmarked edge deployment, and the transformer itself currently runs about 100x too slow for the deadline it targets.
Table of contents
Key takeaways1. The clock that never stops2. Why your favorite LLM runtime breaks the moment you bolt it to a robot3. A camera that never asks permission4. Teaching the brain to say “no”5. The memory problem: forgetting on purpose6. What’s actually running on the GPU7. The honest list of what this is not (yet)8. Where this leaves thingsQuestions this post answers
How can I make an LLM inference runtime respect a hard real-time deadline instead of just computing until done?
Use an admission controller that estimates the cost of the next reasoning chunk with an exponential moving average of prefill and decode times, then refuses to start the chunk if the estimated cost exceeds the remaining deadline budget minus a safety margin. One implementation uses a 33ms deadline with a 2ms safety margin, separate EMAs for prefill (32 tokens) and decode (8 tokens) with alpha 0.2, and falls back to repeating the last action when it declines to admit a chunk. daily.dev surfaces systems-engineering approaches like this for developers building latency-critical inference pipelines.
How should a KV cache decide which frames to evict when memory is full in a vision-language robotics model?
Instead of always evicting the oldest frame (FIFO), a semantic eviction policy computes cosine similarity between pooled embeddings of adjacent retained frames and evicts the older frame from whichever adjacent pair is most similar, keeping frames that capture genuinely new information. In one implementation, each frame costs 32 KV cache slots out of 4096 total, giving room for 128 retained frames before eviction kicks in. developers optimizing memory-constrained inference pipelines can track cache eviction strategies like this on daily.dev.
Why does a chat-oriented LLM serving stack like vLLM or TensorRT-LLM break when connected to a live robot camera feed?
Three failure modes occur simultaneously: VRAM overflows because chat runtimes assume conversations end, but a camera streams tokens forever; control-loop deadlines get missed silently because standard runtimes have no concept of a hard deadline; and frequency mismatch occurs because a 60Hz camera outpaces the reasoning model, forcing perception to either block or accumulate a stale backlog unless something explicitly decouples the two. engineers evaluating inference stacks for real-time robotics workloads can follow this kind of architecture breakdown on daily.dev.