Skip to content
Yardstick
  1. Benchmarks
  2. SWE-Bench Pro (V2 Hard)

SWE-Bench Pro (V2 Hard)

Long, multi-file fixes and features in real repositories, on the subset that still separates frontier coding agents.

CodingRun by Scale AIOfficial leaderboard (opens in a new tab)

Top score
92.2%Claude Fable 5.1
Median of field
88.2%5 models scored
Data as of
Sep 22, 2026Scale AI Labs leaderboard

Standings

Bars share one axis from 0% to 100%.

  1. Rank 1
    Claude Fable 5.192.2%

    Anthropic · $10 / $50New

  2. Rank 2
    GPT-6 Astra90.2%

    OpenAI · $10 / $50New

  3. Rank 3
    Kimi K388.2%

    Moonshot AI · $3 / $15Open weights

  4. Rank 4
    GLM-5.384.3%

    Z.ai · $1.40 / $4.40NewOpen weights

  5. Rank 5
    Gemini 3.8 Flash58.8%

    Google · $0.75 / $3.75New

Not reported for SWE-Bench Pro (V2 Hard) · 24

  • ·
    Claude Haiku 5.5—Not reported

    Anthropic · $0.10 / $0.50New

  • ·
    Claude Opus 5.5—Not reported

    Anthropic · $4 / $20New

  • ·
    Claude Sonnet 5.5—Not reported

    Anthropic · $2 / $10New

  • ·
    DeepSeek-V4-Pro—Not reported

    DeepSeek · $1.32 / $3.96NewOpen weights

  • ·
    DeepSeek-V4.1-Flash—Not reported

    DeepSeek · $0.30 / $1.20NewOpen weights

  • ·
    Gemini 3.1 Pro—Not reported

    Google · $2 / $12Preview

  • ·
    Gemini 3.5 Flash-Lite—Not reported

    Google · $0.30 / $2.50

  • ·
    Gemini 4 Argon—Not reported

    Google · $2 / $10NewPreview

  • ·
    Gemma 4 31B—Not reported

    Google · Self-hostedOpen weights

  • ·
    GLM-5.3-Flash—Not reported

    Z.ai · $0.15 / $0.50NewOpen weights

  • ·
    GPT-6 Luna—Not reported

    OpenAI · $0.10 / $0.50New

  • ·
    GPT-6.1 Sol—Not reported

    OpenAI · $2 / $10New

  • ·
    Grok 4.7—Not reported

    SpaceXAI · $2 / $6New

  • ·
    Kimi K2.7 Code—Not reported

    Moonshot AI · $0.95 / $4Open weights

  • ·
    MiMo-V2.6-Pro—Not reported

    Xiaomi · $0.435 / $0.87NewOpen weights

  • ·
    MiniMax M3—Not reported

    MiniMax · $0.30 / $1.20Open weights

  • ·
    Mistral Large 4—Not reported

    Mistral AI · $1.36 / $4.18NewPreview

  • ·
    Mistral Medium 3.5—Not reported

    Mistral AI · $1.50 / $7.50Open weights

  • ·
    Mistral Small 4—Not reported

    Mistral AI · $0.15 / $0.60Open weights

  • ·
    Muse Glimmer 30B—Not reported

    Meta · Self-hostedNewOpen weights

  • ·
    Muse Spark 1.3—Not reported

    Meta · $1.25 / $4.25New

  • ·
    Qwen3.8-27B—Not reported

    Alibaba · $0.50 / $3NewOpen weights

  • ·
    Qwen3.8-Flash-Next—Not reported

    Alibaba · $0.15 / $0.47NewOpen weights

  • ·
    Qwen3.8-Max—Not reported

    Alibaba · $2 / $6

What it measures

Whether an agent can fix a bug or build a feature in a real codebase so that new tests pass without breaking existing ones. Repositories are chosen to resist training-data leakage.

How it is scored

Percent of tasks resolved on the Hard subset of the V2 public split, with web access disabled and every patch re-graded on a clean image. Each row names its harness (Claude Code, Codex or mini-swe-agent).

Read it with care

On the full V2 set every listed system already scores between 90% and 99%, so only the Hard subset is shown. Scale lists eleven systems so far, and several of the newest models have not been run.

Other benchmarks

  • Artificial Analysis Intelligence Index
  • Epoch Capabilities Index
  • LiveBench
  • Terminal-Bench 4.0
  • Arena WebDev
  • GDPval-AA
  • Vending-Bench 2
  • ARC-AGI-2
  • ARC-AGI-3
  • FrontierMath (Tiers 1–3)
  • MathArena
  • SimpleBench
  • Humanity's Last Exam (Diamond)
  • GPQA Diamond
  • Arena Vision
  • Arena Text
Yardstick

A plain reading of how today’s SI models compare. Scores are taken from each benchmark’s own leaderboard wherever one exists; anything a lab reported about its own model is marked.

Data as of Oct 8, 2026.

Explore

  • Models
  • Compare
  • Benchmarks
  • Value
  • Labs
  • Methodology & sources

Sources

  • Artificial Analysis (opens in a new tab)
  • Arena (opens in a new tab)
  • Epoch AI Benchmarking Hub (opens in a new tab)
  • ARC Prize (opens in a new tab)
  • Terminal-Bench (opens in a new tab)
  • Scale AI Labs (opens in a new tab)

Yardstick is an independent reference and is not affiliated with, sponsored by or endorsed by any lab listed. Product names and logos are trademarks of their respective owners and are used only to identify them. Benchmarks are imperfect proxies: check a model on your own task before you commit to it.

  • Models
  • Compare
  • Benchmarks
  • Value
  • Labs
  • Models
  • Compare
  • Benchmarks
  • Value
  • Labs
  • Models
  • Compare
  • Benchmarks
  • Value
  • Labs