Methodology & benchmarks
A short guide to what the scores mean, where they come from, and the caveats you should keep in mind when picking a model.
Where the numbers come from
Every score on this site is a percentage reported by either the model's vendor in their launch evaluation, or an independent leaderboard. We favor verified, third-party harnesses (Scale AI's SWE-bench Pro, Artificial Analysis, BenchLM) over vendor self-reports where both exist. Where numbers come from different harnesses, treat comparisons as directional, not exact.
The benchmarks
SWE-bench Verified
SWE-Vrf · higher is betterReal GitHub issues, human-verified patches. The standard coding-agent benchmark.
SWE-bench Pro
SWE-Pro · higher is better2,294 harder real-world GitHub issues. Where the frontier separates now.
Terminal-Bench 2.x
Term-B · higher is betterAgentic command-line workflows — planning, iteration, tool coordination.
GPQA Diamond
GPQA · higher is betterGoogle-proof graduate-level science Q&A. General reasoning proxy.
MMLU-Pro
MMLU · higher is betterMulti-task language understanding across 120+ academic subjects.
HumanEval+
HumEv · higher is betterFunctional code generation from docstrings.
MATH-500
MATH · higher is betterCompetition-style math problems.
Humanity's Last Exam
HLE · higher is betterHardest reasoning benchmark (with tools). ~50% = near frontier.
AIME 2025
AIME · higher is betterAmerican Invitational Mathematics Examination.
OSWorld
OSW · higher is betterReal computer-use tasks across desktop apps.
How to read the scores
- Top-end clustering.On saturated benchmarks (GPQA Diamond, SWE-bench Verified), differences of 1–2 points are within noise. Don't over-index on them.
- Hard benchmarks separate the field. SWE-bench Pro, Terminal-Bench 2.x, and HLE are where real gaps show up between frontier models.
- Pricing is per 1M tokensat standard API list price, input / output. “Blended” uses a 3:1 input-to-output ratio as a rough production-workload estimate.
- Context window is the vendor-stated maximum. Usable context (where the model still retrieves accurately) is usually lower.
Caveats
- Vendors sometimes change models silently after launch.
- Benchmark scaffolding (tools, system prompts) changes scores dramatically.
- Agentic benchmarks especially reward harness engineering, not just model capability.
- Always run an eval on your own data before committing to a model in production.