Allen AI is releasing olmo-eval, an open-source evaluation workbench designed for the iterative LLM development loop rather than one-off model scoring. Built on top of their earlier OLMES standard, olmo-eval decouples benchmark logic from runtime policy through a task/suite/harness abstraction, supports agentic and multi-turn evaluations, provides a sandbox and capability-routing layer for tool-using models, and includes a pairwise results viewer for comparing model checkpoints question by question. Unlike tools such as Harbor that target publishable agent benchmarks in sealed containers, olmo-eval prioritizes speed and flexibility during active development — letting teams add benchmarks with minimal code, reuse components across evaluations, and detect whether small performance changes are real improvements or statistical noise using standard error and minimum detectable effect metrics.
