A team shares a seven-month update on their homegrown AI code review workflow, describing a shift toward agent-written code with heavy reliance on AI review to catch missed edge cases. They dropped third-party review tools (like Sentry's AI review beta) in favor of a custom GitHub Action reviewer, made it a blocking merge gate requiring human override with justification, and tackled inconsistency across runs by feeding the bot PR descriptions, prior review comments, and author replies. They also added a local pre-push review step using a Claude skill, and note that switching the underlying model from a hosted commercial one to a self-hosted open-weights model degraded output quality, requiring prompt re-tuning. The overall takeaway: the system only works because it's continuously maintained.
Table of contents
The bigger change is what I stopped doing.We refined our homegrown review.The hard part is consistency.Run it locally first.It’s still a work in progress.Questions this post answers
Why does my AI code review bot give different findings every time I run it on the same pull request?
Inconsistent AI review findings typically stem from each run starting cold with only the diff and no memory of prior context. Feeding the model the PR description, its own earlier review comments, and the author's replies (including override justifications) helps it build on previous findings instead of re-deriving everything from scratch, reducing new must-fixes appearing on unchanged code. Teams debugging flaky AI review output can find similar workflow write-ups on daily.dev.
Does switching from a hosted commercial LLM to a self-hosted open-weights model affect AI code review quality?
Yes, switching the model behind an AI code review pipeline from a hosted commercial model to a self-hosted open-weights model can produce a noticeable step down in quality, with less consistently formatted output and more scattered findings. Prompts tuned for one model do not carry over cleanly to another and require significant rewriting to match the new model's behavior. Anyone weighing hosted versus self-hosted models for coding workflows can track these trade-offs on daily.dev.
How can I make an AI code review bot block pull request merges safely?
A review bot can be turned into a blocking gate by having it post a single comment listing must-fix and should-fix findings, checking for must-fixes with a simple regex, and requesting changes on the PR when any are found. Authors must then address the issues or override the review with a specific label and a justification comment, alongside a required human approver. Developers building enforceable AI review gates can follow similar setups on daily.dev.