The Third Bit: Twelve Ways to Be Wrong About AI-Assisted Coding
This title could be clearer and more informative.Try out Clickbait Shieldfor free (5 uses left this month).
A critical examination of twelve common methodological flaws in studies and claims about AI coding tool productivity. Covers problems including proxy metrics like lines of code, artificial task timing, lack of control groups, self-report bias (Hawthorne effect, novelty effect, social desirability), Goodhart's Law applied to commit counts, measuring only code generation speed while ignoring review burden and security debt, treating adoption rate or suggestion acceptance rate as success metrics, selection bias from volunteer studies, individual vs. system-level measurement, short novelty-period studies, and comparing AI to no-tool baselines rather than existing developer alternatives. Each flaw is backed by academic citations including studies showing AI tools can increase task completion time, introduce security vulnerabilities, and raise code complexity over time.
