<!-- mobian-agent-page publisher="dailydev" canonical="https://daily.dev/posts/4HVeRSnMd" -->

---
title: When you are measuring how intelligent model is, in fact...
description: An experiment comparing GLM-5.3-flash and Grok 4.6 on finding bugs in Django's codebase without being given clues reveals a gap between models that can fix a...
canonical: https://daily.dev/posts/when-you-are-measuring-how-intelligent-model-is-in-fact-you-mostly-measure-how-good-it-was-followin-4hversnmd
twitter:card: summary_large_image
twitter:site: @dailydotdev
og:type: website
og:site_name: daily.dev
og:title: When you are measuring how intelligent model is, in fact you mostly measure how good it was following the plan, or question, based on your own intelligence. Difference is subtle, but crucial. | daily.dev
og:description: An experiment comparing GLM-5.3-flash and Grok 4.6 on finding bugs in Django's codebase without being given clues reveals a gap between models that can fix a...
og:url: https://daily.dev/posts/when-you-are-measuring-how-intelligent-model-is-in-fact-you-mostly-measure-how-good-it-was-followin-4hversnmd
og:image: https://api.daily.dev/og/posts/4HVeRSnMd.png
og:image:alt: When you are measuring how intelligent model is, in fact you mostly measure how good it was following the plan, or question, based on your own intelligence. Difference is subtle, but crucial.
og:image:width: 1200
og:image:height: 630
og:locale: en
---

> ## Documentation Index
> Fetch the complete documentation index at: https://daily.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# When you are measuring how intelligent model is, in fact you mostly measure how good it was following the plan, or question, based on your own intelligence. Difference is subtle, but crucial.

**[AI](https://daily.dev/sources/ai)** · [@leonidbugaev](https://daily.dev/leonidbugaev) · 3 upvotes · 2 comments

## Summary

An experiment comparing GLM-5.3-flash and Grok 4.6 on finding bugs in Django's codebase without being given clues reveals a gap between models that can fix a described bug and models that can independently discover one. Three real Django defects are walked through in detail: a None-handling guard that silently corrupts filenames, a number formatter that inserts a thousands separator inside a scientific-notation exponent, and a deepcopy call that fails on unpicklable objects inside an empty Q() query. GLM-5.3-flash scored zero out of seven on deliberately hard, hand-picked defects across multiple prompt revisions, even when asked for line-by-line execution traces. The piece argues that benchmarks like SWE-bench measure patching a pre-described issue, not discovering an unseen one, and notes OpenAI stopped reporting SWE-bench Verified scores on February 23, 2026, citing contamination and flawed evaluations.

## Full article

daily.dev links to this article rather than hosting it. Read it at the original source: <https://blog.reqproof.com/p/your-ai-can-fix-the-bug-can-it-find>

## Community discussion

Top comments from developers on daily.dev.

**@shoaibmalik2** · 0 upvotes

> Nice , But first try to make it.

## Similar posts on daily.dev

- [Bug Detection Blind Spots in AI Coding Harnesses \(GStack and Beyond\)](https://daily.dev/posts/bug-detection-blind-spots-in-ai-coding-harnesses-gstack-and-beyond--5k8lonjtf) · Towards Data Science · 0 upvotes · 0 comments
- [Coding Models Are Doing Too Much](https://daily.dev/posts/coding-models-are-doing-too-much-ribjn9dws) · Hacker News · 2 upvotes · 1 comments
- [I Gave 4 Local Coding Models the Same FastAPI Bug. The Fastest One Broke Its Own Fix.](https://daily.dev/posts/i-gave-4-local-coding-models-the-same-fastapi-bug-the-fastest-one-broke-its-own-fix--mvnrdlueh) · Medium · 0 upvotes · 0 comments
- [We burned 11.7bn tokens to find the best cyber AI model](https://daily.dev/posts/we-burned-11-7bn-tokens-to-find-the-best-cyber-ai-model-tbgc2dpsf) · Aikido Security · 0 upvotes · 0 comments
- [AI gets good at finding bugs, not as good at fixing them](https://daily.dev/posts/ai-gets-good-at-finding-bugs-not-as-good-at-fixing-them-sptd5ofrp) · The Register · 0 upvotes · 0 comments

---

Tags: [#ai](https://daily.dev/tags/ai), [#llm](https://daily.dev/tags/llm), [#ai-coding](https://daily.dev/tags/ai-coding), [#django](https://daily.dev/tags/django), [#code-review](https://daily.dev/tags/code-review)

[View this post on daily.dev](https://daily.dev/posts/when-you-are-measuring-how-intelligent-model-is-in-fact-you-mostly-measure-how-good-it-was-followin-4hversnmd)

```json
{"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]}
{"@context":"https://schema.org","@type":"DiscussionForumPosting","mainEntityOfPage":"https://daily.dev/posts/when-you-are-measuring-how-intelligent-model-is-in-fact-you-mostly-measure-how-good-it-was-followin-4hversnmd","headline":"When you are measuring how intelligent model is, in fact you mostly measure how good it was following the plan, or question, based on your own intelligence. Difference is subtle, but crucial.","text":"Shared: Your AI can fix the bug. But can it find it?","url":"https://daily.dev/posts/when-you-are-measuring-how-intelligent-model-is-in-fact-you-mostly-measure-how-good-it-was-followin-4hversnmd","datePublished":"2026-09-08T06:48:29.661Z","dateModified":"2026-09-08T06:48:29.665Z","author":{"@type":"Person","name":"Leonid Bugaev","url":"https://daily.dev/leonidbugaev","image":"https://lh3.googleusercontent.com/a/ACg8ocLhMgurwTmJElWaH9A8Ju8cHvZxgMCDt009jEYkmFCyUGAoaGSw=s96-c","description":"Head of Engineering at Tyk.io. Building reqproof.com and probelabs.com.","worksFor":{"@type":"Organization","name":"Tyk","logo":"https://media.daily.dev/image/upload/s--b24KmLW_--/f_auto/v1736353882/companies/tyk"},"interactionStatistic":{"@type":"InteractionCounter","interactionType":{"@type":"EndorseAction"},"userInteractionCount":3320}},"image":"https://media.daily.dev/image/upload/s--ZrL_HSsR--/f_auto/v1722860399/public/Placeholder%2006","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":3},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":2}],"sharedContent":{"@type":"WebPage","url":"https://api.daily.dev/r/euv7luZ9x"},"comment":[{"@type":"Comment","text":"Nice , But first try to make it.","datePublished":"2026-09-13T05:11:34.486Z","url":"https://daily.dev/posts/4HVeRSnMd#c-yoDANlIWo","author":{"@type":"Person","name":"Shoaib Malik","url":"https://daily.dev/shoaibmalik2","image":"https://lh3.googleusercontent.com/a/ACg8ocI8io_KsUa7gVHm8JVbP4fvV97DSHaBjMjAnZ6BmwHXYzh4hyyP=s96-c"}}],"isPartOf":{"@type":"WebPage","url":"https://daily.dev/squads/ai","name":"AI"}}
{"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"AI","item":"https://daily.dev/squads/ai"},{"@type":"ListItem","position":3,"name":"When you are measuring how intelligent model is, in fact you mostly measure how good it was following the plan, or question, based on your own intelligence. Difference is subtle, but crucial."}]}
```

