LangChain built ReviewBench, an internal benchmark for evaluating code review agents using real PR feedback from their LangSmith mono-repo. Rather than synthetic bugs, it uses curated comments from trusted reviewers, filtered through an LLM gate and manual review to isolate substantive, verifiable defects. Tasks are written in Harbor format and scored on F1 combining coverage (did the agent find the real issue?) and precision (were its findings valid?). Current models with a basic harness recover only ~30% of baseline issues. A key finding: prompting strategy matters significantly — adding a structured review prompt to one model (Luna) substantially improved its score without changing tools or the model itself, suggesting that how an agent approaches review is as important as which model powers it.
