Rent a GPU

Devstral Small 24B hardware requirements

23.6B parameters. Needs about 16.2 GB at Q4_K_M with 8K context.

Parameters24BDense
Q4_K_M · 8K16.2 GBthe usual download
FP16 · 8K49.1 GBfull precision
Runs well on18/51GPUs and Macs we track

Memory needed at each quantization

Weights, KV cache and runtime overhead, at 8K and 32K tokens of context.

QuantQuality8K context32K context
FP16Full quality49.1 GB53.2 GB
Q8_0Near-lossless27 GB31 GB
Q6_KExcellent21.3 GB25.3 GB
Q5_K_MVery good18.7 GB22.8 GB
Q4_K_MGood, most popular16.2 GB20.3 GB
Q3_K_MNoticeably degraded13.5 GB17.5 GB
Q2_KHeavily degraded10.8 GB14.8 GB

Run it with Ollama

Downloads Q4_K_M by default.

ollama run devstral:24b

Model card

mistralai/Devstral-Small-2507 on Hugging Face: licence, prompt format and GGUF builds.

Run it in the cloud

Run Devstral Small 24B on a rented GPU today.

It needs about 16.2 GB at Q4_K_M: a rented RTX 3090 / RTX 4090 runs it for about $0.18 to $0.89 an hour.

RunPodRent a GPU on RunPodVast.aiRent a GPU on Vast.ai

Every GPU and Mac checked

Sorted by memory. Bars show how much of each card Devstral Small 24B fills at the best quant that fits.

18 Runs well

NameMemory usedBest quantSpeed

23 Fits with heavy quantization

NameMemory usedBest quantSpeed

10 Too big

NameMemory usedBest quantSpeed

Other Mistral models