--- title: Are LLMs still surprisingly bad at some simple tasks? description: A year-long follow-up experiment tests whether modern LLMs (Gemini Flash and its extended thinking mode, Claude, Perplexity, and an unnamed GPT model) can... canonical: https://daily.dev/posts/are-llms-still-surprisingly-bad-at-some-simple-tasks--4i7vllocg twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Are LLMs still surprisingly bad at some simple tasks? | daily.dev og:description: A year-long follow-up experiment tests whether modern LLMs (Gemini Flash and its extended thinking mode, Claude, Perplexity, and an unnamed GPT model) can... og:url: https://daily.dev/posts/are-llms-still-surprisingly-bad-at-some-simple-tasks--4i7vllocg og:image: https://api.daily.dev/og/posts/4I7VllOCg.png og:image:alt: Are LLMs still surprisingly bad at some simple tasks? og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Are LLMs still surprisingly bad at some simple tasks? **[Terence Eden](https://daily.dev/sources/shkspr)** · 8 min read · 45 upvotes · 13 comments ## Summary A year-long follow-up experiment tests whether modern LLMs (Gemini Flash and its extended thinking mode, Claude, Perplexity, and an unnamed GPT model) can correctly answer which top-level domains share names with valid HTML5 elements. Every model except Siri (which copied a human-written answer verbatim) got it wrong in some way: hallucinating fake TLDs, missing valid matches, or padding answers with irrelevant near-misses to appear thorough. The piece reveals the trick behind the question - custom HTML elements (any hyphenated lowercase tag) and MathML elements vastly expand the valid answer set beyond standard tags, which none of the tested models accounted for. The author argues this exposes a broader problem: people trust LLM output on boring, verifiable tasks precisely because they assume no reasonable mistake could be made, and models increasingly disguise wrongness with confident, verbose non-answers. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### Which top-level domains have the same name as valid HTML5 elements, including custom elements? Over 150 TLDs match valid HTML5 element names once custom elements are counted, since custom element names only need to be lowercase, contain no uppercase letters, and include a dash. Standard elements also match some TLDs directly, like .data for the element, plus .audio, .map, .select, and .search. MathML elements embedded in HTML add mn, mo, ms, and mtr as further TLD matches. _Anyone fact-checking an AI's claims about web standards can cross-reference discussions like this on daily.dev._ ### Why did Google Gemini's default Flash model give wrong answers about HTML elements and TLDs? The default Flash model hallucinated invalid TLDs and elements, listing .a as a TLD (TLDs require at least two characters) and claiming art, app, and bar were valid HTML elements when they never existed as standard tags. Gemini's extended thinking mode avoided fabrication but still missed real matches like the data, map, select, and search elements. _Developers deciding which model to trust for factual lookups can compare real failure cases shared on daily.dev._ ### What are the validity rules for custom HTML elements? A custom HTML element name must start with a lowercase letter, must not contain any uppercase letters, and must contain at least one dash, per the HTML specification's custom element naming rules. This means names like holiday are invalid as custom elements, while dash-containing names like xn--vermgensberatung-pwb are valid, which is why so many Punycode-encoded international TLDs qualify as valid custom element names. _Developers building custom web components can track spec quirks like these through daily.dev._ ## Community discussion Top comments from developers on daily.dev. **@dragonsmako** · 10 upvotes > To only give a short answer to the title: Yes **@zildeus** · 1 upvotes > fun article, thanks. **@ray73864** · 1 upvotes > No they aren't. It's all about the model and how you use it. I've had Claude Code do things for me and it's been impressive with them, I've also had Qwen3.8-Flash-Next do things just as good. > > > But then I've hade the likes of Gemma4 do things, and, it sucked. **@ristotoldsep** · 1 upvotes > Wrong, but with excellent formatting. That's the dangerous bit. **@trevorsuna** · 1 upvotes > Boring questions are dangerous precisely because people skip verification. This example also shows why an answer benchmark needs an explicit ontology: standard HTML elements, custom elements, and MathML produce different valid sets. A model should surface that ambiguity before confidently enumerating results. ## Similar posts on daily.dev - [Researchers discover a shortcoming that makes LLMs less reliable](https://daily.dev/posts/researchers-discover-a-shortcoming-that-makes-llms-less-reliable-wiiiapmcn) · MIT News · 3 upvotes · 0 comments --- Tags: [#webdev](https://daily.dev/tags/webdev), [#llm](https://daily.dev/tags/llm), [#html](https://daily.dev/tags/html), [#claude](https://daily.dev/tags/claude), [#google-gemini](https://daily.dev/tags/google-gemini) [View this post on daily.dev](https://daily.dev/posts/are-llms-still-surprisingly-bad-at-some-simple-tasks--4i7vllocg) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Are LLMs still surprisingly bad at some simple tasks?","url":"https://daily.dev/posts/are-llms-still-surprisingly-bad-at-some-simple-tasks--4i7vllocg","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/are-llms-still-surprisingly-bad-at-some-simple-tasks--4i7vllocg"},"datePublished":"2026-09-22T11:39:32.994Z","dateModified":"2026-09-22T11:39:59.974Z","description":"A year-long follow-up experiment tests whether modern LLMs (Gemini Flash and its extended thinking mode, Claude, Perplexity, and an unnamed GPT model) can...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/5546fa589075a9cb8277f30c15671d82?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/5546fa589075a9cb8277f30c15671d82?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Terence Eden","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Terence Eden","logo":"https://media.daily.dev/image/upload/s--nhylgPuf--/f_auto,q_auto/v1774964413/logos/shkspr?_a=BAMAMiWQ0","url":"https://daily.dev/sources/shkspr"},"commentCount":13,"discussionUrl":"https://daily.dev/posts/are-llms-still-surprisingly-bad-at-some-simple-tasks--4i7vllocg","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":45},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":13}],"keywords":"webdev,llm,html,claude,google-gemini","timeRequired":"PT8M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Terence Eden","item":"https://daily.dev/sources/shkspr"},{"@type":"ListItem","position":3,"name":"Are LLMs still surprisingly bad at some simple tasks?"}]} {"@context":"https://schema.org","@type":"WebPage","@id":"https://daily.dev/posts/are-llms-still-surprisingly-bad-at-some-simple-tasks--4i7vllocg","comment":[{"@type":"Comment","text":"To only give a short answer to the title: Yes","datePublished":"2026-09-22T12:52:34.683Z","url":"https://daily.dev/posts/4I7VllOCg#c-sSybDlS9b","author":{"@type":"Person","name":"DragonsMako","url":"https://daily.dev/dragonsmako","image":"https://media.daily.dev/image/upload/s--O0TOmw4y--/f_auto/v1715772965/public/noProfile"},"interactionStatistic":{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":10}},{"@type":"Comment","text":"fun article, thanks.","datePublished":"2026-09-23T08:01:33.747Z","url":"https://daily.dev/posts/4I7VllOCg#c-d3mKUwDzE","author":{"@type":"Person","name":"ZilDeus","url":"https://daily.dev/zildeus","image":"https://avatars.githubusercontent.com/u/131702027?v=4"},"interactionStatistic":{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":1}},{"@type":"Comment","text":"No they aren’t. It’s all about the model and how you use it. I’ve had Claude Code do things for me and it’s been impressive with them, I’ve also had Qwen3.8-Flash-Next do things just as good.\nBut then I’ve hade the likes of Gemma4 do things, and, it sucked.","datePublished":"2026-09-23T09:00:21.236Z","url":"https://daily.dev/posts/4I7VllOCg#c-OTXoNAytP","author":{"@type":"Person","name":"Ray","url":"https://daily.dev/ray73864","image":"https://avatars.githubusercontent.com/u/1343103?v=4"},"interactionStatistic":{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":1}},{"@type":"Comment","text":"Wrong, but with excellent formatting. That’s the dangerous bit.","datePublished":"2026-09-23T14:16:32.930Z","url":"https://daily.dev/posts/4I7VllOCg#c-wlRekGeJt","author":{"@type":"Person","name":"Risto Tõldsep","url":"https://daily.dev/ristotoldsep","image":"https://lh3.googleusercontent.com/a/ACg8ocLDWc6mZn0JwNmXXw6WY0L_HJ6pegzRttooC5VgtXESHSj3MNYx=s96-c"},"interactionStatistic":{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":1}},{"@type":"Comment","text":"Boring questions are dangerous precisely because people skip verification. This example also shows why an answer benchmark needs an explicit ontology: standard HTML elements, custom elements, and MathML produce different valid sets. A model should surface that ambiguity before confidently enumerating results.","datePublished":"2026-09-24T06:21:32.425Z","url":"https://daily.dev/posts/4I7VllOCg#c-MZ8wTeMfj","author":{"@type":"Person","name":"Trevor Suna","url":"https://daily.dev/trevorsuna","image":"https://media.daily.dev/image/upload/s--dZ7gXxpp--/f_auto/v1784081551/avatars/avatar_EMoP47rpuw8DNjhp6R1b6?_a=BAMAMicg0"},"interactionStatistic":{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":1}}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/are-llms-still-surprisingly-bad-at-some-simple-tasks--4i7vllocg#faq","mainEntity":[{"@type":"Question","name":"Which top-level domains have the same name as valid HTML5 elements, including custom elements?","acceptedAnswer":{"@type":"Answer","text":"Over 150 TLDs match valid HTML5 element names once custom elements are counted, since custom element names only need to be lowercase, contain no uppercase letters, and include a dash. Standard elements also match some TLDs directly, like .data for the element, plus .audio, .map, .select, and .search. MathML elements embedded in HTML add mn, mo, ms, and mtr as further TLD matches. Anyone fact-checking an AI's claims about web standards can cross-reference discussions like this on daily.dev."}},{"@type":"Question","name":"Why did Google Gemini's default Flash model give wrong answers about HTML elements and TLDs?","acceptedAnswer":{"@type":"Answer","text":"The default Flash model hallucinated invalid TLDs and elements, listing .a as a TLD (TLDs require at least two characters) and claiming art, app, and bar were valid HTML elements when they never existed as standard tags. Gemini's extended thinking mode avoided fabrication but still missed real matches like the data, map, select, and search elements. Developers deciding which model to trust for factual lookups can compare real failure cases shared on daily.dev."}},{"@type":"Question","name":"What are the validity rules for custom HTML elements?","acceptedAnswer":{"@type":"Answer","text":"A custom HTML element name must start with a lowercase letter, must not contain any uppercase letters, and must contain at least one dash, per the HTML specification's custom element naming rules. This means names like holiday are invalid as custom elements, while dash-containing names like xn--vermgensberatung-pwb are valid, which is why so many Punycode-encoded international TLDs qualify as valid custom element names. Developers building custom web components can track spec quirks like these through daily.dev."}}]} ```