--- title: Are AI labs pelicanmaxxing? – Dylan Castillo | daily.dev description: An experiment testing whether AI labs are training specifically on Simon Willison's famous 'pelican riding a bicycle' SVG benchmark. The author generated 1,008... canonical: https://daily.dev/posts/are-ai-labs-pelicanmaxxing-dylan-castillo-uc46wcun6 twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Are AI labs pelicanmaxxing? – Dylan Castillo | daily.dev og:description: An experiment testing whether AI labs are training specifically on Simon Willison's famous 'pelican riding a bicycle' SVG benchmark. The author generated 1,008... og:url: https://daily.dev/posts/are-ai-labs-pelicanmaxxing-dylan-castillo-uc46wcun6 og:image: https://api.daily.dev/og/posts/Uc46WCUn6.png og:image:alt: Are AI labs pelicanmaxxing? – Dylan Castillo og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Are AI labs pelicanmaxxing? – Dylan Castillo **[Hacker News](https://daily.dev/sources/hn)** · 11 min read · 0 upvotes · 0 comments ## Summary An experiment testing whether AI labs are training specifically on Simon Willison's famous 'pelican riding a bicycle' SVG benchmark. The author generated 1,008 SVGs across 7 frontier models using a 8×6 grid of animal/vehicle combinations, scored them with an LLM judge, and ran fixed-effects regression to detect any benchmark-specific boost. Results show no statistically significant evidence of pelicanmaxxing: pelicans rank 6th of 8 animals, bicycles rank near last among vehicles, and no lab shows a meaningful boost on the specific pelican-bicycle combination. The one near-signal (GLM-5.2's cell effect) doesn't survive multiple-comparisons correction. The universal right-facing direction in pelican-bicycle images is explained by general tendencies for both pelicans and bicycles. The more plausible concern is general SVGmaxxing across all animal-vehicle combinations, which this experiment cannot detect. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Community take How the wider developer community reacted, aggregated from 1 discussion and 427 comments across hackernews (as of 2026-07-23). **TL;DR:** The community is largely positive about the methodology and findings, appreciating that a quantitative analysis found no evidence of pelican-specific benchmark cheating, while debating whether SVG generation is a genuinely useful capability or a niche benchmark that labs may be broadly optimizing for. **Sentiment:** 55% positive · 35% mixed · 10% skeptical **The case for** - The 8x6 grid methodology across 1008 SVGs is more robust than casual spot-checking and provides real quantitative signal. - SVG generation is genuinely useful for diagrams, icons, charts, CAD, and architecture documentation — not just a toy benchmark. - Models improving at SVG generation appears to correlate with improvements in other spatial/code tasks like OpenSCAD 3D modeling. - The benchmark's absurdity makes it harder to specifically overfit, giving it some validity as a proxy for general capability. **The pushback** - Labs (notably Gemini) have openly discussed SVGmaxxing, so broad SVG optimization rather than pelican-specific cheating can't be ruled out. - The LLM judge used for scoring (single model, no consistency checks) may produce unreliable or random scores. - Top frontier models (Fable 5, GPT-5.6) are now out-pelicaned by lesser models, suggesting the benchmark no longer tracks general capability well. - All pelican-bicycle SVGs face right and place both rider legs on the right side, suggesting models lack genuine understanding of bicycle mechanics. - LLM progress in niche areas may reflect densified training data rather than improved general problem-solving capability. **By community** - hackernews (positive): Commenters broadly praised the methodology and found the no-cheating conclusion credible, while engaging in lively debate about SVGmaxxing, benchmark validity, and the practical utility of LLM SVG generation. **Hottest debate:** Whether labs are specifically SVGmaxxing (broadly optimizing SVG output) rather than pelicanmaxxing, and whether that distinction matters for the benchmark's validity as a general capability signal. **Open questions** - Can SVGmaxxing be distinguished from general code/spatial reasoning improvement, or are they the same thing? - Would testing other output formats (binary, other text formats) reveal similar or different improvement patterns? - How reliable are LLM judges for scoring visual SVG quality, given concerns about consistency? - Does improvement on pelican SVGs still correlate with real-world task performance, or has that relationship broken down? **Highlights** > > The pelican prompt is ridiculous Yes, deliberately so. It was never intended as a meaningful benchmark. The surprising thing was that for the first ~12 months performance on the stupid pelican benchmark did seem to correspond to the performance of the models on other tasks. That pattern no longer holds - Fable 5 and GPT-5.6 have both been out-pelicaned by lesser models now. > — [simonw on hackernews](https://news.ycombinator.com/item?id=49012245) > > All 21 pelican-bicycle images, across all seven labs, face right. No other animal/vehicle combination does that. > However, facing right is common: 60% of all 1,008 images do it. How common depends on the animal and the vehicle, and bicycles are one of the two vehicles where it’s strongest Of course the pelican on the bicycle is facing right. The drivetrain on a bicycle is on the right side. If you want any representation of a bicycle that shows the drivetrain you're going to show the right side of it if you want to do so without the frame occluding it. It's an excellent bet that their training data reflects this. Citation: https://www.rei.com/c/bikes Edited to add: As near as I can tell, all of the bicycles are shown facing right, regardless of the direction the animal is facing (GPT 5.6-Terra, Sample 1/3). Also, in every case where the rider has legs (i.e. not the whale) both of the rider's legs are on the right side of the bicycle. This suggests a pretty serious lack of actual understanding of how a bicycle works. > — [mauvehaus on hackernews · 4 comments](https://news.ycombinator.com/item?id=49012054) > Gemini have absolutely been SVGmaxxing. They've openly talked about it. > — [simonw on hackernews](https://news.ycombinator.com/item?id=49013900) > I use LLMs for 3D CAD design in OpenSCAD. There seems to be a very strong correlation between models that are good at SVG and models that are good at 3D CAD. Anecdote I know, but there does seem to be generalization going on here. > — [nl on hackernews · 1 comments](https://news.ycombinator.com/item?id=49015628) > I agree with the conclusion and am happy to see this blog post, but this killed a bit of credibility for me: > Using a single LLM judge for scoring. Every score here comes from one model, GPT-5.6 Luna, looking at one image at a time. I didn’t do much alignment and didn’t check how often it agrees with itself on a re-run. Having used a similar setup (with previous gen LLMs) to evaluate the 3D models that my product[0] generates, it turned out there was no correlation at all. LLM judgments were very much random and I assume judging SVGs is not that far from judging 3D models. I guess I have to re-test this with current gen. [0]: https://grandpacad.com > — [ponyous on hackernews](https://news.ycombinator.com/item?id=49020810) **Source threads** - [hackernews](https://news.ycombinator.com/item?id=49010129) · 496 points · 427 comments ## Similar posts on daily.dev - [What happens if AI labs train for pelicans riding bicycles?](https://daily.dev/posts/what-happens-if-ai-labs-train-for-pelicans-riding-bicycles--lb0v74rks) · Simon Willison · 2 upvotes · 0 comments - [Agentic Pelican on a Bicycle](https://daily.dev/posts/agentic-pelican-on-a-bicycle-6v5vgmwem) · Hacker News · 1 upvotes · 0 comments - [Qwen3.6-35B-A3B on my laptop drew me a better pelican than Claude Opus 4.7](https://daily.dev/posts/qwen3-6-35b-a3b-on-my-laptop-drew-me-a-better-pelican-than-claude-opus-4-7-htpbdbpvw) · Simon Willison · 8 upvotes · 0 comments --- Tags: [#llm](https://daily.dev/tags/llm) [View this post on daily.dev](https://daily.dev/posts/are-ai-labs-pelicanmaxxing-dylan-castillo-uc46wcun6) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Are AI labs pelicanmaxxing? – Dylan Castillo","url":"https://daily.dev/posts/are-ai-labs-pelicanmaxxing-dylan-castillo-uc46wcun6","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/are-ai-labs-pelicanmaxxing-dylan-castillo-uc46wcun6"},"datePublished":"2026-07-22T19:13:45.546Z","dateModified":"2026-07-23T13:54:03.851Z","description":"An experiment testing whether AI labs are training specifically on Simon Willison's famous 'pelican riding a bicycle' SVG benchmark. The author generated 1,008...","image":"https://dylancastillo.co/posts/images/social_media_card.webp","thumbnailUrl":"https://dylancastillo.co/posts/images/social_media_card.webp","isAccessibleForFree":true,"articleSection":"Hacker News","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Hacker News","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/hn","url":"https://daily.dev/sources/hn"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/are-ai-labs-pelicanmaxxing-dylan-castillo-uc46wcun6","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"llm","timeRequired":"PT11M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Hacker News","item":"https://daily.dev/sources/hn"},{"@type":"ListItem","position":3,"name":"Are AI labs pelicanmaxxing? – Dylan Castillo"}]} ```