Open models vs Claude, GPT and Gemini
The newest open-weight models now trade blows with the best paid models on the makers' own benchmarks. Here are the published scores, side by side, and what it takes to run each one yourself.
| Benchmark | Kimi K3.1 Pro | Claude Fable 5.1 | GPT-6 Astra | Claude Opus 5.5 |
|---|---|---|---|---|
| Terminal-Bench 4.0coding agent in a terminal | 65.7 | 57.9 | 58.2 | 66.4 |
| SWE Marathon v1.1long software tasks | 51.7 | 45.6 | 42.5 | 46.3 |
| DeepSWE v1.1fixing real code | 75.5 | 67.4 | 74.1 | 74.2 |
| HLE-Full w/ toolsexpert questions, with tools | 70.0 | 65.6 | 57.2 | 67.7 |
| BrowseCompweb research | 92.1 | — | 91.5 | 90.8 |
| Benchmark | GLM-5.3 | Claude Opus 4.8 | Claude Fable 5 | GPT-5.6 Sol |
|---|---|---|---|---|
| Terminal Bench 2.1coding agent in a terminal | 88.2 | 85.0 | 88.0 | 88.8 |
| DeepSWE v1.1fixing real code | 66.9 | 58.0 | 69.7 | 72.7 |
| CyberGymsecurity tasks | 84.5 | 78.1 | 83.8 | 83.6 |
| Toolathlon Verifiedusing tools and apps | 73.0 | 76.2 | 74.7 | 74.9 |
| HLE w/ Toolsexpert questions, with tools | 62.5 | 57.9 | 63.9 | 64.5 |
| Benchmark | DeepSeek V4.1 Flash | Claude Opus 5.0 | GPT-5.6 Sol |
|---|---|---|---|
| GPQA Diamondgraduate-level science | 90.9 | 93.4 | 94.1 |
| HLEexpert questions | 36.8 | 56.3 | 44.5 |
| Terminal-Bench 2.1coding agent in a terminal | 90.6 | 89.1 | 88.8 |
| DeepSWE v1.1fixing real code | 74.2 | 74.0 | 73.0 |
| NL2Repo-Benchwriting a whole repository | 64.0 | 75.3 | 56.8 |
| Benchmark | DeepSeek V4 Pro (0813) | Claude Opus 4.8 | Claude Fable 5 |
|---|---|---|---|
| HLE w/ toolsexpert questions, with tools | 60.0 | 57.9 | 63.0 |
| Terminal Bench 2.1coding agent in a terminal | 87.9 | 85.0 | 88.0 |
| DeepSWEfixing real code | 62.7 | 58.0 | 70.0 |
| Toolathlon-Verifiedusing tools and apps | 74.1 | 76.2 | 77.9 |
Scores are for Qwen3.8-Max, Qwen's hosted version of these open weights.
| Benchmark | Qwen3.8-Max | Claude Opus 4.8 | Claude Fable 5 | GPT-5.6 Sol |
|---|---|---|---|---|
| Terminal Bench 2.1coding agent in a terminal | 86.6 | 84.6 | 84.6 | 88.8 |
| SWE-bench Profixing real code | 67.7 | 69.2 | 80.0 | 64.6 |
| GPQA Diamondgraduate-level science | 92.6 | 92.0 | 92.6 | 94.1 |
| HLEexpert questions | 43.6 | 45.7 | 53.3 | 47.2 |
| Benchmark | GLM-5 | Claude Opus 4.5 | Gemini 3 Pro | GPT-5.2 |
|---|---|---|---|---|
| SWE-bench Verifiedfixing real code | 77.8 | 80.9 | 76.2 | 80.0 |
| GPQA-Diamondgraduate-level science | 86.0 | 87.0 | 91.9 | 92.4 |
| HLEexpert questions | 30.5 | 28.4 | 37.2 | 35.4 |
| BrowseCompweb research | 62.0 | 37.0 | 37.8 | — |
Fits on one 24 GB graphics card.
| Benchmark | Qwen3.8 27B | Claude Opus 4.6 (max) |
|---|---|---|
| Terminal Bench 2.1coding agent in a terminal | 73.0 | 78.2 |
| SWE-bench Profixing real code | 61.7 | 53.4 |
| GPQA Diamondgraduate-level science | 89.2 | 91.3 |
| LiveCodeBench v6coding problems | 90.3 | 88.8 |
| HLEexpert questions | 30.8 | 40.0 |
What it takes to run them yourself
Memory at Q4_K_M with 8K context, the smallest cloud setup that holds it, and RunPod / Vast.ai prices. None of these fit on a home PC; the API column is the same model hosted by someone else, per million input / output tokens on OpenRouter.
| Model | Memory | Cloud setup | Per hour | API $/M tokens |
|---|---|---|---|---|
 Kimi K3.1 Pro 2.8T-A104B | 1698.3 GB | 8× B300 | $71.92/hr | $2 / $10 |
 Qwen3.8 2.4T-A95B | 1484.3 GB | 8× B300 | $71.92/hr | $2 / $6 |
 DeepSeek V4 Pro 1.6T-A49B | 971.2 GB | 8× H200 | $42.32/hr | $0.117 / $4.66 |
 Mistral Large 4 1T-A52B | 607.5 GB | 8× A100 / H100 | $8.88 to $31.92/hr | $0.7 / $2.1 |
| DeepSeek V4.1 Flash | 463.4 GB | 8× A100 / H100 | $8.88 to $31.92/hr | $0.051 / $0.567 |
 GLM-5.3 | 457.8 GB | 8× A100 / H100 | $8.88 to $31.92/hr | $0.07 / $1.4 |
| GLM-5 744B-A40B | 452.4 GB | 8× A100 / H100 | $8.88 to $31.92/hr | $1 / $3.2 |
 MiniMax M3 428B-A23B | 260.5 GB | 4× A100 / H100 | $4.44 to $15.96/hr | $0.3 / $1.2 |
| GLM-5.3 Flash 320B-A18B | 195.3 GB | 4× A100 / H100 | $4.44 to $15.96/hr | $0.15 / $0.5 |
| Qwen3.8 Flash Next | 109.9 GB | H200 | $5.29/hr | $0.15 / $0.47 |
Close to the frontier, on your own card
You don't need a server for a taste of this. These newer mid-size models run on a single 24 to 32 GB graphics card or a 32 GB Mac.
Frequently asked questions
Are open models as good as Claude or ChatGPT now?
On the benchmarks their makers publish, the largest open models (Kimi K3.1 Pro, GLM-5.3, DeepSeek V4, Qwen3.8) score close to, and on some tests above, the paid frontier models. Makers choose which tests and rivals to show, so treat any single table as a claim, not a verdict.
Can I run GLM-5 or Kimi K3.1 at home?
Not on a normal PC. GLM-5 needs about 450 GB of memory at Q4_K_M and Kimi K3.1 Pro about 1.7 TB, which means renting a server with eight data-centre GPUs, or using the model through an API.
What is the best open model I can run on one GPU?
Among the models we track, Qwen3.8 27B and Gemma 4 31B fit on a single 24 to 32 GB card at Q4_K_M. Qwen reports Qwen3.8 27B scoring 89.2 on GPQA Diamond against 91.3 for Claude Opus 4.6.