Meta's efficient 70B instruction model that matches or exceeds Llama 3.1 70B across key benchmarks with improved inference efficiency.
| Benchmark | Description | Score | Category | Bar |
|---|---|---|---|---|
| MMLU | Massive Multitask Language Understanding | 86.1% | knowledge | |
| HumanEval | Code generation accuracy | 88.4% | coding | |
| MATH | Mathematical reasoning | 87.8% | math | |
| GSM8K | Grade school math | 91.5% | math | |
| GPQA | Graduate-level science Q&A | 47.8% | reasoning | |
| HellaSwag | Common sense reasoning | 86.5% | reasoning | |
| SWE-bench | Software engineering problems | 35.2% | coding | |
| ARC-C | AI2 Reasoning Challenge | 94% | reasoning |