--- title: Can Frontier Models See Past a Decoy? Elorian Tests... description: A new benchmark called DecoyCount tests whether AI models truly count objects or rely on visual shortcuts. Frontier models all scored below 50% accuracy, with... canonical: https://daily.dev/posts/can-frontier-models-see-past-a-decoy-elorian-tests-visual-reasoning-zv5y7m5dn twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Can Frontier Models See Past a Decoy? Elorian Tests Visual Reasoning | daily.dev og:description: A new benchmark called DecoyCount tests whether AI models truly count objects or rely on visual shortcuts. Frontier models all scored below 50% accuracy, with... og:url: https://daily.dev/posts/can-frontier-models-see-past-a-decoy-elorian-tests-visual-reasoning-zv5y7m5dn og:image: https://api.daily.dev/og/posts/Zv5y7m5Dn.png og:image:alt: Can Frontier Models See Past a Decoy? Elorian Tests Visual Reasoning og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Can Frontier Models See Past a Decoy? Elorian Tests Visual Reasoning **[Latent Space](https://daily.dev/sources/latentspace)** · 6 min read · 0 upvotes · 0 comments ## Summary A new benchmark called DecoyCount tests whether AI models truly count objects or rely on visual shortcuts. Frontier models all scored below 50% accuracy, with GPT-6 Astra highest at 43.17%, followed by Gemini-3.8-flash at 32.71%, Qwen3.8-max at 28.31%, and Claude-Fable-5.1 at 25.22%. Elorian's new visual reasoning model, Glimpse, scored 50.48% in an internal test, the only model to clear 50%, though its result came from a single run rather than the three-run average used for the other models. Elorian, which raised $55 million in seed funding, argues visual reasoning requiring models to generate and edit images as part of their reasoning trace is a foundational step toward AGI. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### What score did GPT-6 Astra get on the DecoyCount visual counting benchmark? GPT-6 Astra scored 43.17% on DecoyCount, the highest among frontier models tested, all of which fell below 50% accuracy. Gemini-3.8-flash scored 32.71%, Qwen3.8-max scored 28.31%, and Claude-Fable-5.1 scored 25.22%. These results were averaged across three runs each under their reported settings, according to the accompanying paper. _Comparing model benchmarks before picking one? Track vision model evaluations like these on daily.dev._ ### What is the DecoyCount benchmark and why was it created? DecoyCount is a benchmark co-released by Elorian and Mercor that tests whether language models genuinely count objects or rely on visual shortcuts and substitution heuristics, such as miscounting a mirror reflection as a separate object. It contains 1,660 items and evaluates vision and reasoning together, since most benchmarks only check the final number and can't tell whether an error is a vision problem or a reasoning problem. _Evaluating whether a model actually reasons rather than guesses? Follow benchmark breakdowns like this on daily.dev._ ### How did Elorian's Glimpse model score on the DecoyCount benchmark compared to frontier models? Glimpse scored 50.48% on the 1,660 DecoyCount items, making it the only model to clear 50%, about 7 points ahead of GPT-6 Astra's 43.17%. However, Glimpse's score came from a single run without an agent or tool harness, while the six frontier models were each averaged across three runs under their reported settings, a methodological difference worth noting. _Weighing a new visual reasoning model against established frontier models? Catch comparisons like this on daily.dev._ ## Similar posts on daily.dev - [AI Model Comparison](https://daily.dev/posts/ai-model-comparison-xyyfl4rqa) · Hacker News · 1 upvotes · 0 comments - [AI models get better at math but still get low marks](https://daily.dev/posts/ai-models-get-better-at-math-but-still-get-low-marks-bugdv2bcl) · The Register · 1 upvotes · 0 comments --- Tags: [#deep-learning](https://daily.dev/tags/deep-learning), [#world-models](https://daily.dev/tags/world-models) [View this post on daily.dev](https://daily.dev/posts/can-frontier-models-see-past-a-decoy-elorian-tests-visual-reasoning-zv5y7m5dn) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Can Frontier Models See Past a Decoy? Elorian Tests Visual Reasoning","url":"https://daily.dev/posts/can-frontier-models-see-past-a-decoy-elorian-tests-visual-reasoning-zv5y7m5dn","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/can-frontier-models-see-past-a-decoy-elorian-tests-visual-reasoning-zv5y7m5dn"},"datePublished":"2026-10-28T21:48:24.861Z","dateModified":"2026-10-28T22:24:37.420Z","description":"A new benchmark called DecoyCount tests whether AI models truly count objects or rely on visual shortcuts. Frontier models all scored below 50% accuracy, with...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/7aaefc990010a56bc484736c682e769c?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/7aaefc990010a56bc484736c682e769c?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Latent Space","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Latent Space","logo":"https://media.daily.dev/image/upload/s--DnJ9laFj--/f_auto,q_auto/v1780213196/logos/latentspace?_a=BAMAMiWQ0","url":"https://daily.dev/sources/latentspace"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/can-frontier-models-see-past-a-decoy-elorian-tests-visual-reasoning-zv5y7m5dn","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"deep-learning,world-models","timeRequired":"PT6M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Latent Space","item":"https://daily.dev/sources/latentspace"},{"@type":"ListItem","position":3,"name":"Can Frontier Models See Past a Decoy? Elorian Tests Visual Reasoning"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/can-frontier-models-see-past-a-decoy-elorian-tests-visual-reasoning-zv5y7m5dn#faq","mainEntity":[{"@type":"Question","name":"What score did GPT-6 Astra get on the DecoyCount visual counting benchmark?","acceptedAnswer":{"@type":"Answer","text":"GPT-6 Astra scored 43.17% on DecoyCount, the highest among frontier models tested, all of which fell below 50% accuracy. Gemini-3.8-flash scored 32.71%, Qwen3.8-max scored 28.31%, and Claude-Fable-5.1 scored 25.22%. These results were averaged across three runs each under their reported settings, according to the accompanying paper. Comparing model benchmarks before picking one? Track vision model evaluations like these on daily.dev."}},{"@type":"Question","name":"What is the DecoyCount benchmark and why was it created?","acceptedAnswer":{"@type":"Answer","text":"DecoyCount is a benchmark co-released by Elorian and Mercor that tests whether language models genuinely count objects or rely on visual shortcuts and substitution heuristics, such as miscounting a mirror reflection as a separate object. It contains 1,660 items and evaluates vision and reasoning together, since most benchmarks only check the final number and can't tell whether an error is a vision problem or a reasoning problem. Evaluating whether a model actually reasons rather than guesses? Follow benchmark breakdowns like this on daily.dev."}},{"@type":"Question","name":"How did Elorian's Glimpse model score on the DecoyCount benchmark compared to frontier models?","acceptedAnswer":{"@type":"Answer","text":"Glimpse scored 50.48% on the 1,660 DecoyCount items, making it the only model to clear 50%, about 7 points ahead of GPT-6 Astra's 43.17%. However, Glimpse's score came from a single run without an agent or tool harness, while the six frontier models were each averaged across three runs under their reported settings, a methodological difference worth noting. Weighing a new visual reasoning model against established frontier models? Catch comparisons like this on daily.dev."}}]} ```