A controlled experiment compares Kimi K3's 1M-token context window against a RAG pipeline on the same 12 questions over a 127,068-token personal corpus. Long-context answered all 12 questions completely, while RAG scored lower mainly on completeness (0.83/2) despite matching on groundedness. Long-context cost 16x more and took roughly 3x longer per question, and the entry-tier daily quota (1.5M tokens) was broken by a single 12-question long-context run. The piece also details three failure modes: reasoning tokens silently consuming the completion budget and producing empty answers, non-deterministic runs since Kimi K3 only allows temperature=1, and an unreliable prefix cache (only 33% hit rate) that made real costs land closer to the worst-case estimate. Recommendation: use long-context for small, rarely-queried corpora, and RAG once query volume scales up.
Table of contents
1 — Why the question looks different now2 — The setup: One corpus, two paths3 — 12 questions in three difficulty levels4 — Evaluation: Why I graded blind5 — Three things that went wrong6 – Results7 – When I would use whichFinal ThoughtsWhere To Continue Learning?Questions this post answers
Why did my Kimi K3 API calls return completely empty answers even though there was no error?
Kimi K3 is a reasoning model whose internal thinking tokens count against the same max_completion_tokens limit as the visible answer. If the model uses the entire budget thinking, finish_reason reports 'length' and the answer field is empty, even though the call otherwise looks successful in logs. Raising max_completion_tokens (e.g., from 800 to 4000) and logging finish_reason surfaces and fixes the issue. daily.dev surfaces practical debugging writeups like this for developers wiring up reasoning models in production.
Is RAG actually cheaper and faster than dumping a full document corpus into a long-context LLM prompt?
For a small corpus (127,068 tokens, about 12% of a 1M-token window) queried only a handful of times, long-context won on completeness (12/12 full scores vs RAG's 0.83/2 on completeness) despite costing about 16 times more and taking roughly 3 times longer per question. The economics flip at scale: for 12,000 queries, projected costs were roughly $3,800 (long-context) versus $230 (RAG), favoring RAG once query volume is high. track cost and latency trade-offs like these when deciding between RAG and long-context on daily.dev.
Why does Moonshot AI's prefix cache for Kimi K3 not reliably reduce input token costs on repeated long-context calls?
In testing, the prefix cache only kicked in on 3 of 8 identical calls with the same 127,346-token corpus prefix, with no discernible pattern, and it expired entirely after a five-day gap, showing 0 cached tokens on a later call. A cached call cost $0.0466 versus $0.3916 without the cache, roughly 8x cheaper, so unreliable caching can make real-world costs land close to the uncached worst case, about 30% higher than optimistic estimates. daily.dev helps developers planning LLM costs stay aware of quirks like unreliable prefix caching.
319 Impressions1 Comment