Skip to main content
Agentic AI Hub
Get the app

Benchmarks & Leaderboards

Table of Contents
Link copied!

The single most useful mental model here is that benchmarks tell you what a model can do on someone else's task. Only your own eval tells you what it does on yours. Read benchmarks to shortlist, then test on your workload (LLMOps: Evals, Observability & Guardrails).

The benchmarks worth knowing (and their catch)

LMArena / Chatbot Arena. Human preference: users vote between two anonymous answers, and the votes become Elo ratings (the chess rating system, where beating a stronger opponent raises your score more). Good for: a real-world read on open-ended chat quality and which model people actually prefer. Catch: rewards length and formatting (use the style-controlled board), has no ground truth, and small rank gaps are noise. → lmarena.ai

MMLU / MMLU-Pro. Broad multiple-choice knowledge across dozens of subjects. Good for: a rough general-knowledge floor. Catch: the original is saturated (frontier models cluster in the low 90s) and contaminated, so use MMLU-Pro (harder, 10 options). → MMLU-Pro board

GPQA Diamond. Expert science reasoning, 198 PhD-level "Google-proof" questions. Good for: telling genuine scientific reasoning apart from recall. Catch: approaching saturation, and with only 198 items a few questions swing the score.

SWE-bench Verified. Real coding: fix actual GitHub issues so the project's hidden tests pass (quote the 500-item verified subset). Good for: the most credible single "can it fix real bugs" number. Catch: heavily scaffold/harness-dependent (the same model varies widely by agent), Python-centric, and older instances risk contamination (see SWE-bench Pro). → swebench.com

Terminal-Bench. Agentic CLI competence: multi-step tasks in a real sandboxed shell. Good for: how well an agent actually drives a terminal, close to real agent work. Catch: small task count, and the harness matters as much as the model. → tbench.ai

τ²-bench (tau-bench). Tool use across multi-turn conversations against a stateful database under policy rules. Good for: agent reliability, since its pass^k metric (solved on all k runs) exposes brittleness that single runs hide. Catch: narrow domains, and results depend on the simulated user. → github

AIME / FrontierMath. Math reasoning. Good for: hard, checkable step-by-step reasoning. FrontierMath (Epoch AI) keeps its problems private and still has real headroom. Catch: AIME problems leak every year, so always check which exam year is quoted. → FrontierMath

Humanity's Last Exam (HLE). Frontier knowledge, ~2,500 expert questions built to stay hard. Good for: separating the very top models (scores are still only in the low-mid 20s%). Catch: pure knowledge and reasoning, no agentic ability or tool use.

HumanEval. Legacy code-gen (164 Python functions). Good for: almost nothing now, it's saturated (96-98%) and contaminated. Use LiveCodeBench instead (problems released after training cutoffs). Also worth knowing: GDPval (professional real-work tasks), Aider Polyglot (multi-language edits), GAIA (general assistant), and SimpleBench (common-sense traps where humans beat models).

Live leaderboards to bookmark

  • Artificial Analysis. Intelligence index + live speed, latency, price, context in one view. Best for cost/performance trade-offs.
  • LMArena. Human-preference Elo with category/style boards. Best for subjective chat quality.
  • llm-stats.com. 300+ models, composite score with price baked in. Best for fast head-to-head selection.
  • LM Council. Aggregates ~18 independently-run benchmarks. Best for avoiding vendor self-reports.
  • Vals AI. Industry-specific evals (legal, finance, healthcare). Best for domain buyers.
  • OpenRouter Rankings. Real token-usage by model/app across a large routing marketplace. Best for "what devs actually ship."
  • Vercel AI Gateway leaderboards. Models ranked by real usage through Vercel's gateway. Best for the app-builder / TypeScript perspective.
  • Scale SEAL / Scale Labs. Private, contamination-resistant expert evals (incl. SWE-bench Pro). Best for trust when public benchmarks are gamed.
  • Vellum LLM Leaderboard. Clean side-by-side benchmark + price/context tables. Best for a quick reference sheet.
  • SWE-bench / Terminal-Bench. The canonical coding-agent boards.

Task-specific arenas (LMArena spin-offs and others, when you care about one skill): WebDev Arena (web-app building), Copilot/Code Arena (in-editor coding), Design Arena (UI/design generation), Search Arena (retrieval-grounded answers), Vision/Text-to-Image Arena (multimodal). Each isolates one capability the general boards blur together.

Why leaderboards mislead (keep these in mind)

  • Contamination. Public benchmarks leak into training data, so a high score can be memorization. Prefer contamination-resistant or refreshed evals.
  • Saturation. Once models cluster near the ceiling, the gaps between them are noise.
  • Style and gaming. Preference boards reward length and formatting, and vendors can tune to a benchmark's format.
  • Harness dependence. Agentic scores depend as much on the scaffolding as the model, so compare same-harness numbers.

(Papers with Code shut down in 2025, so don't rely on it for current SOTA.)

Link copied!