Can the AMD Radeon RX 7900 XTX run Mistral Small 24B?
- FP16
- Q8_0
- Q6_K
- Q5_K_M
- Q4_K_M
- Q3_K_M
- Q2_K
Yes. The AMD Radeon RX 7900 XTX can run Mistral Small 24B fully in VRAM at Q6_K (excellent), using about 21.3 GB of the 24 GB available, at roughly 30 tokens/second.
Run it with Ollama
With Ollama installed, one command downloads and starts Mistral Small 24B:
ollama run mistral-small:24bOllama's default download is Q4_K_M (about 16.2 GB here). You have room for Q6_K, which is excellent: pick that GGUF in LM Studio or from Hugging Face.
Run it in the cloud
No AMD Radeon RX 7900 XTX? Run Mistral Small 24B in the cloud.
Rent a cloud GPU by the hour. No upfront cost, stop any time.
Memory needed by quantization
At 8,192 tokens of context. The black line marks the 24 GB the AMD Radeon RX 7900 XTX can use.
| Quant | Quality | Memory | Fits? | Speed | |
|---|---|---|---|---|---|
| FP16 | Full quality | 49.1 GB | Too big | — | |
| Q8_0 | Near-lossless | 27 GB | Too big | — | |
| Q6_K | Excellent | 21.3 GB | Fits | ~30 tok/s | |
| Q5_K_M | Very good | 18.7 GB | Fits | ~34 tok/s | |
| Q4_K_M | Good, most popular | 16.2 GB | Fits | ~40 tok/s | |
| Q3_K_M | Noticeably degraded | 13.5 GB | Fits | ~50 tok/s | |
| Q2_K | Heavily degraded | 10.8 GB | Fits | ~65 tok/s |
GPUs that run Mistral Small 24B comfortably
Frequently asked questions
Can AMD Radeon RX 7900 XTX run Mistral Small 24B?
Yes. The AMD Radeon RX 7900 XTX can run Mistral Small 24B fully in VRAM at Q6_K (excellent), using about 21.3 GB of the 24 GB available, at roughly 30 tokens/second.
How much VRAM does Mistral Small 24B need?
About 16.2 GB at Q4_K_M with 8,192 tokens of context, and about 49.1 GB at full FP16 precision.
How fast is Mistral Small 24B on the AMD Radeon RX 7900 XTX?
Roughly 30 tokens per second at Q6_K, estimated from the AMD Radeon RX 7900 XTX's 960 GB/s memory bandwidth.