Google's original most capable model, designed to be a general-purpose AI that outperforms experts on 30 of 32 benchmarks.
| Benchmark | Description | Score | Category | Bar |
|---|---|---|---|---|
| MMLU | Massive Multitask Language Understanding | 90% | knowledge | |
| HumanEval | Code generation accuracy | 74.4% | coding | |
| MATH | Mathematical reasoning | 94.4% | math | |
| GSM8K | Grade school math | 94.4% | math | |
| GPQA | Graduate-level science Q&A | 59.4% | reasoning | |
| HellaSwag | Common sense reasoning | 87.2% | reasoning | |
| SWE-bench | Software engineering problems | 26.9% | coding | |
| ARC-C | AI2 Reasoning Challenge | 96% | reasoning |