Can the NVIDIA RTX 5090 run Mixtral 8x7B?
Yes
QuantQ4_K_MGood, most popular
Memory30 GBof 32 GB
Speed~137tokens/s
Yes. The NVIDIA RTX 5090 can run Mixtral 8x7B fully in VRAM at Q4_K_M (good, most popular), using about 30 GB of the 32 GB available, at roughly 137 tokens/second.
Try other context lengths in the calculator →Memory needed by quantization
At 8,192 tokens of context.
| Quant | Quality | Memory | Fits? | Speed |
|---|---|---|---|---|
| FP16 | Full quality | 95.1 GB | Too big | — |
| Q8_0 | Near-lossless | 51.3 GB | Too big | — |
| Q6_K | Excellent | 40 GB | Too big | — |
| Q5_K_M | Very good | 34.9 GB | Too big | — |
| Q4_K_M | Good, most popular | 30 GB | Fits | ~137 tok/s |
| Q3_K_M | Noticeably degraded | 24.5 GB | Fits | ~171 tok/s |
| Q2_K | Heavily degraded | 19.2 GB | Fits | ~222 tok/s |
How to run Mixtral 8x7B on the NVIDIA RTX 5090
With Ollama installed, one command downloads and starts it:
ollama run mixtral:8x7bOllama's default download is Q4_K_M (about 30 GB here). That is also the best quant that fits.
GPUs that run Mixtral 8x7B comfortably
Models that run well on the NVIDIA RTX 5090
Run it in the cloud
Don't want to buy hardware? Run Mixtral 8x7B on a rented GPU today.
Rent a cloud GPU by the hour. No upfront cost, cancel any time.