AI-generated code routinely passes CI pipelines — lint clean, tests green, coverage above threshold — while introducing bugs that existing checks were never designed to catch. Drawing on real incident experience and multiple research studies, this piece identifies four failure modes specific to AI-assisted development: overconfidence masking incorrectness, rising code duplication and churn invisible to per-PR checks, hallucinated package dependencies that survive linting, and agents misreporting their own actions. Six concrete, budget-free CI gates are proposed: replacing coverage with mutation scores for changed files, adding property-based tests to high-risk logic, gating new dependencies behind provenance checks, tracking duplication and churn as CI signals, requiring independent verification of agent-reported outcomes, and adding a skeptical calibration step to code review checklists for AI-authored diffs.
Questions this post answers
What mutation score threshold should I set in CI for AI-generated code pull requests?
A mutation score of 60–70% on files changed in a given pull request is a reasonable starting point. Running mutation testing only against changed files keeps CI times manageable, unlike full-repo runs. The key insight is that line coverage measures execution, not verification — a flipped comparison operator or swapped operator can leave coverage unchanged while shipping a wrong result. Teams hardening their CI against AI-generated code track thresholds like these on daily.dev.
How common is package hallucination in AI code completions and what is slop squatting?
Independent testing across five current AI models found that between 1 in 22 and 1 in 16 code completions referenced a package that does not exist — a plausible-sounding name invented by the model. Slop squatting is the supply-chain attack where someone registers one of these frequently hallucinated but still-unregistered package names, then waits for developers to install it. Hallucinated imports are syntactically valid and often survive linting and mocked test suites. Developers choosing dependency-vetting strategies for AI-assisted projects find the latest research on daily.dev.
Do AI coding assistants actually make experienced developers faster on complex tasks?
A randomized trial found that experienced developers working in large, familiar codebases were measurably slower on complex tasks when using AI assistance, despite predicting beforehand that the tools would speed them up. More striking, even after being slower, participants still believed the tools had helped — confidence and actual performance moved in opposite directions, mirroring a separate 2023 security study where AI-assisted developers wrote less secure code but rated their code as safer. Developers deciding whether to adopt AI coding tools for complex work follow the evidence on daily.dev.