--- title: GitHub - cactus-compute/cactus-hybrid: On-device models... description: Cactus Hybrid introduces post-trained on-device models that emit a confidence score (0–1) alongside every answer, enabling hybrid routing: run inference... canonical: https://daily.dev/posts/github---cactus-compute-cactus-hybrid-on-device-models-that-know-when-they-re-wrong-every-answer-c-8jzlzy82o twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: GitHub - cactus-compute/cactus-hybrid: On-device models that know when they're wrong: every answer carries a confidence score for cloud handoff. Copy-paste quickstarts for Cactus, Transformers, llama. | daily.dev og:description: Cactus Hybrid introduces post-trained on-device models that emit a confidence score (0–1) alongside every answer, enabling hybrid routing: run inference... og:url: https://daily.dev/posts/github---cactus-compute-cactus-hybrid-on-device-models-that-know-when-they-re-wrong-every-answer-c-8jzlzy82o og:image: https://api.daily.dev/og/posts/8JzlzY82o.png og:image:alt: GitHub - cactus-compute/cactus-hybrid: On-device models that know when they're wrong: every answer carries a confidence score for cloud handoff. Copy-paste quickstarts for Cactus, Transformers, llama. og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # GitHub - cactus-compute/cactus-hybrid: On-device models that know when they're wrong: every answer carries a confidence score for cloud handoff. Copy-paste quickstarts for Cactus, Transformers, llama. **[Hacker News](https://daily.dev/sources/hn)** · 4 min read · 0 upvotes · 0 comments ## Summary Cactus Hybrid introduces post-trained on-device models that emit a confidence score (0–1) alongside every answer, enabling hybrid routing: run inference locally and only hand off to a larger cloud model when confidence falls below a threshold. The first release is Gemma 4 E2B Hybrid, which matches Gemini 3.1 Flash-Lite on most benchmarks while routing only 15–35% of queries to the cloud. The confidence probe achieves a mean AUROC of 0.814 across text, vision, and audio benchmarks — notably outperforming token entropy and generalizing to audio despite zero audio training data. Copy-paste quickstarts are provided for Cactus, MLX, Hugging Face Transformers, and llama.cpp. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Community take How the wider developer community reacted, aggregated from 1 discussion and 52 comments across hackernews (as of 2026-07-24). **TL;DR:** The community is cautiously interested in the approach of using hidden-state probes to give small on-device LLMs self-awareness about their own errors, but raises pointed philosophical and technical objections about whether a model can truly 'know when it's wrong' versus merely detecting inconsistency or uncertainty. **Sentiment:** 45% positive · 35% mixed · 20% skeptical **The case for** - Freezing main weights while training only added probe weights avoids degrading the base model's quality. - Using rolling token entropy for early exit combined with a probe on complete output is a pragmatic dual-signal approach. - Mechanistic interpretability signals (e.g., shallow vs. deep layer activations) may offer a principled way to distinguish memorized facts from context-derived or doom-looped outputs. **The pushback** - A model cannot truly 'know when it's wrong'—it can only detect inconsistency or uncertainty, and high confidence does not imply correctness. - The confidence score's own calibration is unverified; conformal prediction or multi-method ensembling was suggested but not yet applied. - Re-running the model for consistency checks—a natural validation approach—is impractical on edge devices due to battery drain. - The philosophical catch-all category for 'reasoned judgements' is problematic, as complex compounded assertions are prone to circular reasoning and other fallacies. **By community** - hackernews (mixed): Commenters are genuinely intrigued by the mechanistic probe approach but push back hard on the language of 'knowing when it's wrong' and ask for stronger calibration methodology and broader benchmarks. **Hottest debate:** Whether a model can meaningfully 'know when it's wrong' or can only detect uncertainty/inconsistency, and whether the confidence score itself is trustworthy. **Open questions** - How is the confidence score itself calibrated, and could conformal prediction provide distribution-free guarantees on the threshold? - Does combining token entropy with verbal confidence reporting improve AUROC over either method alone? - How does the probe perform on larger models (e.g., 7B+) and on coding-specific tasks with fallback to a larger model? - What do the full mechanistic study results look like, and when will the report be published? **Highlights** > > "post-trained to know when it's wrong" Is it also post-trained to know when it's wrong about when it's wrong? > "Every response comes with a confidence score between 0 and 1" How confident is it in its confidence? Please, I'm sure that what you're doing is very neat and useful, but use other language to describe it. I beg you. You can't know when you're wrong. You can only know when you're unsure or inconsistent. You can be absolutely certain and still wrong and uncertain and still correct. > — [BugsJustFindMe on hackernews · 4 comments](https://news.ycombinator.com/item?id=49015694) > A few years ago, before RLVR became ubiquitous, people were using generate -> classify loops to get models to improve math reasoning. Turns out classification is sometimes easier to do and more accurate than generating coherent traces. So you'd get a model to "think step by step", then run each step (.split('/n/n')) through a classifier, and tell the model "this step might be wrong, re-check". If you're interested I could probably dig up some papers on this. Now there might be even better ways of getting feedback, I saw some interesting trials using "J-spaces" (from Anthropic's latest research in this area). Turns out you can kinda see when the model uses deep seeded concepts in early layers, or it's "riffing on an idea" with shallow signals in those same layers. And youc an get some signal out of it, and have it re-do a step. It's also worth mentioning that LMs are wrong in different ways from humans. It's not impossible that we'll eventually find ways of better understanding this (through mechanistic interpretation or something else) and have easy "classifiers" for "this comes from the training data" vs. "this comes from early context" (i.e. supplied materials from prompts/discovery) vs. "this comes from later tokens, likely the model doom-looped itself". > — [NitpickLawyer on hackernews · 1 comments](https://news.ycombinator.com/item?id=49018206) > Cool idea, some feedback: 1. Consider using conformal prediction to calibrate the cutoff. Conformal prediction provides a distribution-free guarantee under exchangeability. This would let you turn your raw probe score into a threshold with a guaranteed bound on the rate of wrongly-kept on-device answers. Source: https://en.wikipedia.org/wiki/Conformal_prediction 2. The best indicators of confidence in ML come from multiple independent methods. What was the result if you combine the token entropy and verbal confidence reporting methods? Does this improve the result? 3. I noticed you didn't mention the assessment method of rerunning the model and judging whether outputs are consistent. How does that method compare in terms of AUROC? > — [dmrivers on hackernews · 1 comments](https://news.ycombinator.com/item?id=49019286) > This is an old problem in philosophy, one that the pre-LLM systems were interested in, which got completely bypassed by the present approach. What is knowledge? It's not a simple question even for humans, we don't have a magic truth detector in the brain. https://plato.stanford.edu/entries/knowledge-analysis/ This is why LLMs are more successful in programming than other disciplines: it's easier to assess correctness of results, through proofs, typechecking, and testing. Similarly for maths and applied maths. Much harder in the humanities. > — [pjc50 on hackernews · 1 comments](https://news.ycombinator.com/item?id=49019018) > no because we freeze the main weights and only train our added weights. > — [HenryNdubuaku on hackernews · 1 comments](https://news.ycombinator.com/item?id=49018333) **Source threads** - [hackernews](https://news.ycombinator.com/item?id=49010782) · 122 points · 52 comments ## Similar posts on daily.dev - [Cactus v1: Cross-Platform LLM Inference on Mobile with Zero Latency and Full Privacy](https://daily.dev/posts/cactus-v1-cross-platform-llm-inference-on-mobile-with-zero-latency-and-full-privacy-etbzbhxay) · InfoQ · 1 upvotes · 0 comments - [Empowering app developers: Fine-tuning Gemma 3 for mobile with Tunix in Google Colab](https://daily.dev/posts/empowering-app-developers-fine-tuning-gemma-3-for-mobile-with-tunix-in-google-colab-0wxznzr3k) · Google Open Source Blog · 0 upvotes · 0 comments - [Needle 2 - The 14 MB Agentic LLM for Tiny Devices](https://daily.dev/posts/needle-2---the-14-mb-agentic-llm-for-tiny-devices-eu1fgqsim) · Hacker News · 4 upvotes · 0 comments --- Tags: [#ai-gateway](https://daily.dev/tags/ai-gateway), [#ai-inference](https://daily.dev/tags/ai-inference), [#gemma](https://daily.dev/tags/gemma), [#local-ai](https://daily.dev/tags/local-ai), [#shell](https://daily.dev/tags/shell) [View this post on daily.dev](https://daily.dev/posts/github---cactus-compute-cactus-hybrid-on-device-models-that-know-when-they-re-wrong-every-answer-c-8jzlzy82o) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"GitHub - cactus-compute/cactus-hybrid: On-device models that know when they're wrong: every answer carries a confidence score for cloud handoff. Copy-paste quickstarts for Cactus, Transformers, llama.","url":"https://daily.dev/posts/github---cactus-compute-cactus-hybrid-on-device-models-that-know-when-they-re-wrong-every-answer-c-8jzlzy82o","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/github---cactus-compute-cactus-hybrid-on-device-models-that-know-when-they-re-wrong-every-answer-c-8jzlzy82o"},"datePublished":"2026-07-23T01:55:37.120Z","dateModified":"2026-10-14T02:33:16.104Z","description":"Cactus Hybrid introduces post-trained on-device models that emit a confidence score (0–1) alongside every answer, enabling hybrid routing: run inference...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/4d53e8a36e239dbf558b06c2b5462bbf?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/4d53e8a36e239dbf558b06c2b5462bbf?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Hacker News","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Hacker News","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/hn","url":"https://daily.dev/sources/hn"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/github---cactus-compute-cactus-hybrid-on-device-models-that-know-when-they-re-wrong-every-answer-c-8jzlzy82o","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"ai-gateway,ai-inference,gemma,local-ai,shell","timeRequired":"PT4M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Hacker News","item":"https://daily.dev/sources/hn"},{"@type":"ListItem","position":3,"name":"GitHub - cactus-compute/cactus-hybrid: On-device models that know when they're wrong: every answer carries a confidence score for cloud handoff. Copy-paste quickstarts for Cactus, Transformers, llama."}]} ```