Rent a GPU

Can the NVIDIA RTX 3070 run Phi-3.5 Mini 3.8B?

RTX 3070 × Phi-3.5 Mini 3.8B8K context
YesRuns fully on the GPU
Memory at Q8_07.9 GB of 8 GB
Weights 4.1 GBKV cache 3.2 GBRuntime 0.6 GB
QuantQ8_0Near-lossless
Memory7.9 GBof 8 GB usable
Speed~66tokens per second
  1. FP16
  2. Q8_0
  3. Q6_K
  4. Q5_K_M
  5. Q4_K_M
  6. Q3_K_M
  7. Q2_K

Yes. The NVIDIA RTX 3070 can run Phi-3.5 Mini 3.8B fully in VRAM at Q8_0 (near-lossless), using about 7.9 GB of the 8 GB available, at roughly 66 tokens/second.

Try other context lengths

Run it with Ollama

With Ollama installed, one command downloads and starts Phi-3.5 Mini 3.8B:

ollama run phi3.5:3.8b

Ollama's default download is Q4_K_M (about 6.1 GB here). You have room for Q8_0, which is near-lossless: pick that GGUF in LM Studio or from Hugging Face.

Run it in the cloud

No NVIDIA RTX 3070? Run Phi-3.5 Mini 3.8B in the cloud.

Rent a cloud GPU by the hour. No upfront cost, stop any time.

Rent a GPU on RunPodRent a GPU on Vast.ai

Memory needed by quantization

At 8,192 tokens of context. The black line marks the 8 GB the NVIDIA RTX 3070 can use.

QuantQualityMemoryFits?Speed
FP16Full quality11.5 GBToo big—
Q8_0Near-lossless7.9 GBFits~66 tok/s
Q6_KExcellent7 GBFits~86 tok/s
Q5_K_MVery good6.5 GBFits~99 tok/s
Q4_K_MGood, most popular6.1 GBFits~116 tok/s
Q3_K_MNoticeably degraded5.7 GBFits~144 tok/s
Q2_KHeavily degraded5.3 GBFits~188 tok/s

GPUs that run Phi-3.5 Mini 3.8B comfortably

Models that run well on the NVIDIA RTX 3070

Frequently asked questions

Can NVIDIA RTX 3070 run Phi-3.5 Mini 3.8B?

Yes. The NVIDIA RTX 3070 can run Phi-3.5 Mini 3.8B fully in VRAM at Q8_0 (near-lossless), using about 7.9 GB of the 8 GB available, at roughly 66 tokens/second.

How much VRAM does Phi-3.5 Mini 3.8B need?

About 6.1 GB at Q4_K_M with 8,192 tokens of context, and about 11.5 GB at full FP16 precision.

How fast is Phi-3.5 Mini 3.8B on the NVIDIA RTX 3070?

Roughly 66 tokens per second at Q8_0, estimated from the NVIDIA RTX 3070's 448 GB/s memory bandwidth.