SWE-bench Pro (public)Benchmark scores and sources
Published result; benchmark version and evaluation conditions remain in the id and result note.
| Row | Model | Result | Normalized (0–100) | Confidence |
|---|---|---|---|---|
| 1 | GPT-5.4 OpenAI | 59.1%Official model cards via models.devLab-reported; metric resolve rate; transcribed by MIT models.dev catalog; not independently evaluated [variant] public
Retrieved Oct 9, 2026 · factual citation; MIT transcription Open source ↗ | 59.1 | 100% confidence 100 percent, Full |
| 2 | Gemini 3.5 Flash Google | 55.1%Official model cards via models.devLab-reported; metric resolve rate; transcribed by MIT models.dev catalog; not independently evaluated [variant] single attempt; publicPublished May 19, 2026
Retrieved Oct 9, 2026 · factual citation; MIT transcription Open source ↗ | 55.1 | 100% confidence 100 percent, Full |
| 3 | Claude Opus 4.6 Anthropic | 51.9%Official model cards via models.devLab-reported; metric resolve rate; transcribed by MIT models.dev catalog; not independently evaluated [variant] public
Retrieved Oct 9, 2026 · factual citation; MIT transcription Open source ↗ | 51.9 | 100% confidence 100 percent, Full |
| 4 | Muse Spark 1.1 Meta | 51.5%SWE-bench Pro (public)Published steward score [variant] Published Jul 9, 2026
Retrieved Oct 9, 2026 · factual citation Open source ↗ | 51.5 | 61% confidence 61 percent, Medium |
| 5 | Claude Opus 4.6 Anthropic | 47.1%SWE-bench Pro (public)Published steward score [variant] Published Apr 8, 2026
Retrieved Oct 9, 2026 · factual citation Open source ↗ | 47.1 | 100% confidence 100 percent, Full |
| 6 | Gemini 3.1 Pro Preview Google | 46.1%Official model cards via models.devLab-reported; metric resolve rate; transcribed by MIT models.dev catalog; not independently evaluated [variant] public
Retrieved Oct 9, 2026 · factual citation; MIT transcription Open source ↗ | 46.1 | 100% confidence 100 percent, Full |
| 7 | Claude Opus 4.5 Anthropic | 45.9%Official model cards via models.devLab-reported; metric resolve rate; transcribed by MIT models.dev catalog; not independently evaluated [variant] public
Retrieved Oct 9, 2026 · factual citation; MIT transcription Open source ↗ | 45.9 | 100% confidence 100 percent, Full |
| 8 | Claude Sonnet 4.5 (latest) Anthropic | 43.6%Official model cards via models.devLab-reported; metric resolve rate; transcribed by MIT models.dev catalog; not independently evaluated [variant] public
Retrieved Oct 9, 2026 · factual citation; MIT transcription Open source ↗ | 43.6 | 43% confidence 43 percent, Low |
| 9 | Claude Sonnet 4.5 (latest) Anthropic | 43.6%SWE-bench Pro (public)Published steward score [variant] Published Sep 19, 2025
Retrieved Oct 9, 2026 · factual citation Open source ↗ | 43.6 | 43% confidence 43 percent, Low |
| 10 | GPT-5.4 OpenAI | 43.4%SWE-bench Pro (public)Published steward score [variant] Published Apr 8, 2026
Retrieved Oct 9, 2026 · factual citation Open source ↗ | 43.4 | 100% confidence 100 percent, Full |
| 11 | Gemini 3 Pro Preview Google | 43.3%Official model cards via models.devLab-reported; metric resolve rate; transcribed by MIT models.dev catalog; not independently evaluated [variant] public
Retrieved Oct 9, 2026 · factual citation; MIT transcription Open source ↗ | 43.3 | 68% confidence 68 percent, Medium |
| 12 | Gemini 3 Pro Preview Google | 43.3%SWE-bench Pro (public)Published steward score [variant] Published Nov 26, 2025
Retrieved Oct 9, 2026 · factual citation Open source ↗ | 43.3 | 68% confidence 68 percent, Medium |
| 13 | Claude Sonnet 4 (latest) Anthropic | 42.7%Official model cards via models.devLab-reported; metric resolve rate; transcribed by MIT models.dev catalog; not independently evaluated [variant] public
Retrieved Oct 9, 2026 · factual citation; MIT transcription Open source ↗ | 42.7 | 6% confidence 6 percent, Low |
| 14 | GPT-5 OpenAI | 41.8%Official model cards via models.devLab-reported; metric resolve rate; transcribed by MIT models.dev catalog; not independently evaluated [variant] public
Retrieved Oct 9, 2026 · factual citation; MIT transcription Open source ↗ | 41.8 | 100% confidence 100 percent, Full |
| 15 | GPT-5 OpenAI | 41.8%SWE-bench Pro (public)Published steward score [variant] Published Nov 26, 2025
Retrieved Oct 9, 2026 · factual citation Open source ↗ | 41.8 | 100% confidence 100 percent, Full |
| 16 | GPT-5.2 Codex OpenAI | 41.0%Official model cards via models.devLab-reported; metric resolve rate; transcribed by MIT models.dev catalog; not independently evaluated [variant] public
Retrieved Oct 9, 2026 · factual citation; MIT transcription Open source ↗ | 41.0 | 43% confidence 43 percent, Low |
| 17 | GPT-5.2 Codex OpenAI | 41.0%SWE-bench Pro (public)Published steward score [variant] Published Jan 27, 2026
Retrieved Oct 9, 2026 · factual citation Open source ↗ | 41.0 | 43% confidence 43 percent, Low |
| 18 | Claude Haiku 4.5 (latest) Anthropic | 39.5%Official model cards via models.devLab-reported; metric resolve rate; transcribed by MIT models.dev catalog; not independently evaluated [variant] public
Retrieved Oct 9, 2026 · factual citation; MIT transcription Open source ↗ | 39.5 | 33% confidence 33 percent, Low |
| 19 | Qwen3-Coder 480B-A35B Instruct Alibaba / Qwen | 38.7%Official model cards via models.devLab-reported; metric resolve rate; transcribed by MIT models.dev catalog; not independently evaluated [variant] public
Retrieved Oct 9, 2026 · factual citation; MIT transcription Open source ↗ | 38.7 | 47% confidence 47 percent, Low |
| 20 | MiniMax-M2.1 MiniMax | 36.8%Official model cards via models.devLab-reported; metric resolve rate; transcribed by MIT models.dev catalog; not independently evaluated [variant] public
Retrieved Oct 9, 2026 · factual citation; MIT transcription Open source ↗ | 36.8 | 13% confidence 13 percent, Low |
| 21 | Gemini 3 Flash Preview Google | 34.6%Official model cards via models.devLab-reported; metric resolve rate; transcribed by MIT models.dev catalog; not independently evaluated [variant] public
Retrieved Oct 9, 2026 · factual citation; MIT transcription Open source ↗ | 34.6 | 93% confidence 93 percent, High |
| 22 | Gemini 3 Flash Preview Google | 34.6%SWE-bench Pro (public)Published steward score [variant] Published Jan 12, 2026
Retrieved Oct 9, 2026 · factual citation Open source ↗ | 34.6 | 93% confidence 93 percent, High |
| 23 | GPT-5.2 OpenAI | 29.9%Official model cards via models.devLab-reported; metric resolve rate; transcribed by MIT models.dev catalog; not independently evaluated [variant] public
Retrieved Oct 9, 2026 · factual citation; MIT transcription Open source ↗ | 29.9 | 100% confidence 100 percent, Full |
| 24 | GPT-5.2 OpenAI | 29.9%SWE-bench Pro (public)Published steward score [variant] Published Jan 27, 2026
Retrieved Oct 9, 2026 · factual citation Open source ↗ | 29.9 | 100% confidence 100 percent, Full |
| 25 | Claude Opus 4.5 Anthropic | 23.4%SWE-bench Pro (public)Published steward score [variant] Published Jan 12, 2026
Retrieved Oct 9, 2026 · factual citation Open source ↗ | 23.4 | 100% confidence 100 percent, Full |
| 26 | Qwen3 235B-A22B Alibaba / Qwen | 21.4%Official model cards via models.devLab-reported; metric resolve rate; transcribed by MIT models.dev catalog; not independently evaluated [variant] public
Retrieved Oct 9, 2026 · factual citation; MIT transcription Open source ↗ | 21.4 | 76% confidence 76 percent, Medium |
| 27 | Qwen3 235B-A22B Alibaba / Qwen | 21.4%SWE-bench Pro (public)Published steward score [variant] Published Jan 27, 2026
Retrieved Oct 9, 2026 · factual citation Open source ↗ | 21.4 | 76% confidence 76 percent, Medium |
| 28 | Claude Opus 4.1 (latest) Anthropic | 17.8%SWE-bench Pro (public)Published steward score [variant] Published Sep 19, 2025
Retrieved Oct 9, 2026 · factual citation Open source ↗ | 17.8 | 14% confidence 14 percent, Low |
| 29 | GPT OSS 120B OpenAI | 16.2%SWE-bench Pro (public)Published steward score [variant] Published Jan 27, 2026
Retrieved Oct 9, 2026 · factual citation Open source ↗ | 16.2 | 79% confidence 79 percent, Medium |
| 30 | Gemma 3 27B IT Google | 11.4%SWE-bench Pro (public)Published steward score [variant] Published Jan 27, 2026
Retrieved Oct 9, 2026 · factual citation Open source ↗ | 11.4 | 74% confidence 74 percent, Medium |
| 31 | GLM-4.6 Z.ai | 9.7%Official model cards via models.devLab-reported; metric resolve rate; transcribed by MIT models.dev catalog; not independently evaluated [variant] public
Retrieved Oct 9, 2026 · factual citation; MIT transcription Open source ↗ | 9.7 | 55% confidence 55 percent, Medium |
| 32 | GLM-4.6 Z.ai | 9.7%SWE-bench Pro (public)Published steward score [variant] Published Jan 27, 2026
Retrieved Oct 9, 2026 · factual citation Open source ↗ | 9.7 | 55% confidence 55 percent, Medium |
| 33 | Claude Sonnet 4 Anthropic | 9.1%SWE-bench Pro (public)Published steward score [variant] Published Sep 19, 2025
Retrieved Oct 9, 2026 · factual citation Open source ↗ | 9.1 | 100% confidence 100 percent, Full |
| 34 | Llama 4 Maverick 17B Instruct Meta | 5.2%Official model cards via models.devLab-reported; metric resolve rate; transcribed by MIT models.dev catalog; not independently evaluated [variant] public
Retrieved Oct 9, 2026 · factual citation; MIT transcription Open source ↗ | 5.2 | 78% confidence 78 percent, Medium |
| 35 | Llama 4 Maverick 17B Instruct Meta | 5.2%SWE-bench Pro (public)Published steward score [variant] Published Jan 27, 2026
Retrieved Oct 9, 2026 · factual citation Open source ↗ | 5.2 | 78% confidence 78 percent, Medium |
| 36 | GPT-4o OpenAI | 3.6%SWE-bench Pro (public)Published steward score [variant] Published Sep 19, 2025
Retrieved Oct 9, 2026 · factual citation Open source ↗ | 3.6 | 59% confidence 59 percent, Medium |
Results are as published by the source behind each value (hover or tap the number). Benchmark scores are shown individually for every source/variant (a model may have multiple rows), and vary by version, harness and date; the normalized column uses the method’s fixed 0–100 scales and feeds the coding pillar of the SI Score.