Best Local LLM for 5090 (RTX 5090, 32GB VRAM): 2026 Picks
An RTX 5090 local LLM setup runs Qwen 3.6 35B-A3B at Q6 (~28 GB, ~80 tok/sec) or gpt-oss 20B at Q8 for OpenClaw. The RTX 5090 jumped the consumer LLM ceiling from 24 GB to 32 GB VRAM and nearly doubled memory bandwidth (1008 → 1792 GB/s) over the RTX 4090. That's enough headroom to run Qwen 3.6 27B at Q8 (near-FP16) with 64K context, or step up to MoE models with 35B+ parameters.
Just bought an RTX 5090?
See our AI training options. We'll set up OpenClaw + Ollama to run all your AI locally on the 5090, free.
The RTX 5090's 32 GB is the new-card pick for local LLMs — more headroom than a 24 GB 4090/3090 for bigger context and models. Need 70B at long context? Step up to the 96 GB Blackwell.
Updated August 8, 2026: Gemma 4 31B added — the 5090’s 32 GB is the first consumer card that holds the dense Gemma at Q6_K (~25 GB), a quant 24 GB cards can’t fit.
Shopping, not just reading? Our full gear ladder lists every verified pick by tier — Macs, single GPUs, and multi-GPU builds — with what to buy and what to skip.
RTX 5090 local LLM: the direct answer
For an RTX 5090 local LLM setup, run Qwen 3.6 35B-A3B at Q6_K. It uses about 28 GB of the card’s 32 GB and generates near 80 tok/sec, because only 3B parameters are active per token and the 5090 moves 1792 GB/s. For OpenClaw agent loops, run gpt-oss 20B at Q8_0 instead — the tool-call JSON is cleaner over long unattended runs.
Bottom Line
- Best overall pick: Qwen 3.6 35B-A3B (MoE) at Q6_K — ~80 tok/sec, 35B-class quality
- Best dense model / 256K context: Gemma 4 31B at Q6_K (~25 GB, multimodal)
- Best agentic coding: Laguna XS 2.1 at Q4_K_M (20 GB — 33B MoE, 3B active, 70.9% SWE-Bench Verified)
- Best for OpenClaw production: gpt-oss 20B at Q8_0 (cleanest tool calls)
- Best premium 27B: Qwen 3.6 27B at Q8_0 (near-FP16)
- Best squeeze for 70B: Llama 3.3 70B at Q3_K_S (fits, but quality compromised)
Top Picks for RTX 5090 (32 GB VRAM, 1792 GB/s bandwidth)
1. Qwen 3.6 35B-A3B (Q6_K) — best overall
Mixture-of-Experts variant of Qwen 3.6 (April 22, 2026). 35B total params, 3B active per token. At Q6_K uses about 28 GB. The 5090’s bandwidth + MoE design = blistering inference.
ollama pull qwen3.6:35b-q6_K openclaw config set agents.defaults.models.chat ollama/qwen3.6:35b-q6_K
Expected speed: 75-90 tokens/sec.
2. Laguna XS 2.1 (Q4_K_M) — best agentic coding
Poolside’s July 2, 2026 release. 33B total params, 3B active per token (MoE), 256K native context, 70.9% on SWE-Bench Verified. laguna-xs-2.1:q4_K_M is 20 GB — on a 24 GB card you must cap context at 8-16K, but the 5090’s 32 GB leaves ~12 GB for KV cache, so you can run roughly 64K context. Skip Q8: those weights are ~33 GB and do not fit.
ollama pull laguna-xs-2.1 openclaw config set agents.defaults.models.chat ollama/laguna-xs-2.1
Expected speed: 45-60 tokens/sec (3B active params + 1792 GB/s bandwidth).
3. Gemma 4 31B (Q6_K) — best dense model, 256K context
Google’s dense Gemma 4, 30.7B params across 60 layers with a 256K context window and multimodal text + image input. Q4_K_M is 18.3 GB and Q6_K is 25.2 GB, so the 5090 is the first consumer card that holds the near-lossless quant with KV cache left over — a 24 GB 4090 has to drop to Q4 or run the sparse 26B-A4B instead. Native structured tool use makes it usable for agent loops.
ollama pull gemma4:31b-q6_K openclaw config set agents.defaults.models.chat ollama/gemma4:31b-q6_K
Expected speed: 45-55 tokens/sec.
4. gpt-oss 20B (Q8_0) — best for OpenClaw production
OpenAI’s 20B at full Q8 uses about 22 GB. Cleanest tool-call JSON of any open-weight model.
ollama pull gpt-oss:20b-q8_0 openclaw config set agents.defaults.models.chat ollama/gpt-oss:20b-q8_0 openclaw run --agent --max-hours 8 "Implement the spec end-to-end"
5. Qwen 3.6 27B (Q8_0) — premium quality
Full Q8 of the April 22 release uses about 30 GB with 32K context. Near-FP16 quality. Speed: ~45 tok/sec.
6. Mistral Small 4 (119B-A6B MoE, IQ3_XS) — premium reasoning squeeze
Mistral’s March 16, 2026 release. 119B total params, 6B active. At IQ3_XS uses about 30 GB. Quality is degraded at IQ3 but the underlying model is premium tier.
7. Qwen 3.5 122B-A10B (IQ2_XXS) — biggest squeeze
For breadth of knowledge over inference quality. ~30 GB at IQ2_XXS. Note: Qwen 3.5 has the Ollama tool-calling bug — pair with gpt-oss for agent loops.
What Fits in 32 GB VRAM (RTX 5090)
| Model | Quant | VRAM | Tok/sec |
|---|---|---|---|
| Qwen 3.6 35B-A3B (MoE) | Q6_K | ~28 GB | 75-90 |
| Gemma 4 31B (dense) ← 256K ctx | Q6_K | ~25 GB | 45-55 |
| Laguna XS 2.1 (MoE) ← coding | Q4_K_M | ~20 GB | 45-60 |
| Qwen 3.6 27B | Q8_0 | ~30 GB | 40-50 |
| gpt-oss 20B | Q8_0 | ~22 GB | 70-85 |
| Mistral Small 4 (119B-A6B) | IQ3_XS | ~30 GB | 50-65 (MoE) |
| Llama 3.3 70B | Q3_K_S | ~28 GB | 15-22 (degraded) |
OpenClaw Setup on RTX 5090
ollama pull qwen3.6:35b-q6_K ollama pull gpt-oss:20b-q8_0 openclaw config set agents.defaults.models.chat ollama/qwen3.6:35b-q6_K openclaw config set agents.defaults.models.agent ollama/gpt-oss:20b-q8_0 openclaw config set agents.defaults.keep_alive 30m
Common Mistakes on RTX 5090
- Running Llama 3.3 70B at IQ2 because it fits. Quality at IQ2 is so degraded that Qwen 3.6 27B at Q8 beats it on every benchmark and runs 2-3x faster.
- Maxing context to 256K. KV cache at 256K eats 20+ GB. Cap at 64K-128K depending on the model.
- Pulling Laguna XS 2.1 at Q8 because you have 32 GB. The Q8 weights are ~33 GB — they do not fit before you count KV cache. Use Q4_K_M (20 GB) and spend the headroom on context instead.
- Buying the 5090 just for tokens/sec. The real value is the 32 GB VRAM ceiling. If you only run 24GB-and-under models, the 4090 is half the price and still fast.
🛒 Mac alternative
Want 32GB+ unified memory without the GPU build? Mac Studio Ultra delivers.
Amazon affiliate links — we earn a small commission at no cost to you.
FAQ
What is the best local LLM for an RTX 5090?
Qwen 3.6 35B-A3B (MoE) at Q6_K is the best pick that takes advantage of the 5090's 32 GB VRAM. It uses about 28 GB with comfortable context and runs at ~80 tokens/sec thanks to the MoE design + 1792 GB/s bandwidth. For agentic coding, Laguna XS 2.1 at Q4_K_M (20 GB, 33B MoE with 3B active params, 70.9% SWE-Bench Verified) is the 2026 pick. For OpenClaw, gpt-oss 20B at Q8_0 (~22 GB) is the safer production pick.
Can the RTX 5090 run Laguna XS 2.1?
Yes, comfortably. laguna-xs-2.1:q4_K_M is 20 GB, so the 5090's 32 GB VRAM leaves about 12 GB of headroom — enough KV cache for roughly 64K context, far more than the 8-16K cap a 24 GB card forces. The Q8 weights (~33 GB) do not fit 32 GB, so stay on Q4. Because only 3B parameters are active per token, generation runs at small-model speed despite the 33B total size.
Can the RTX 5090 run Gemma 4 31B?
Yes, and the 5090 is the first consumer card that runs it well. Gemma 4 31B is Google's dense 30.7B model with a 256K context window; Q4_K_M is about 18.3 GB and Q6_K is about 25.2 GB, so the 5090's 32 GB fits Q6_K with room left for KV cache. That is the quant a 24 GB 4090 cannot hold. On 24 GB cards the sparse Gemma 4 26B-A4B (25.2B total, 3.8B active, ~16.9 GB at Q4_K_M) is the Gemma pick instead.
RTX 5090 vs RTX 4090 — is the upgrade worth it for LLMs?
Yes if you want to step past 24GB. The 5090 unlocks Qwen 3.6 27B at Q8 quality, Qwen 3.6 35B-A3B at Q6, or DeepSeek V3 at heavy quants. On 24GB-and-under workloads, the 5090 is ~60% faster than the 4090 (1792 vs 1008 GB/s bandwidth) but the cost premium is steep. Buy the 5090 for the VRAM ceiling, not just the speed.
Can the RTX 5090 run a 70B model?
Yes at degraded quants. Llama 3.3 70B at Q3_K_S uses about 28 GB — fits the 5090 with KV cache for short context. Quality is degraded vs Q4 but workable. For full Q4 70B you still need 48 GB+, so either two 5090s or a workstation GPU.
What is the best model to run on a 5090?
Qwen 3.6 35B-A3B at Q6_K. It uses about 28 GB of the 32 GB VRAM and generates 75-90 tokens/sec, because only 3B parameters are active per token. For agentic coding, run Laguna XS 2.1 at Q4_K_M (20 GB). For OpenClaw production loops, run gpt-oss 20B at Q8_0 (22 GB).
Is the RTX 5090 good for local LLM use?
Yes. Its 32 GB VRAM and 1792 GB/s bandwidth make it the fastest single consumer card for local LLMs. It holds Qwen 3.6 27B at Q8_0 (~30 GB) and Gemma 4 31B at Q6_K (~25 GB), quants a 24 GB card cannot fit. Buy it for the 32 GB ceiling, not only for tokens/sec.
What is the best LLM for 32GB VRAM?
Qwen 3.6 35B-A3B at Q6_K (~28 GB) for general use and speed. Qwen 3.6 27B at Q8_0 (~30 GB) gives the highest 27B quality at about 45 tok/sec. Llama 3.3 70B fits only at Q3_K_S (~28 GB) with short context, so a 70B at Q4 still needs 48 GB or more. See the 32GB VRAM setup guide for the card options.
Before you order parts, check the tested hardware list for current prices by tier.
See Also
- RTX 5090 vs RTX 4090 vs Used RTX 3090 — whether the 32GB ceiling is worth paying for
- Best Local LLM for RTX 4090 — same family, 24GB tier
- Best Local LLM for RTX A6000 → — 48GB workstation
- Mac Studio vs RTX Workstation for Local LLMs — whether to buy Apple unified memory or NVIDIA CUDA
- Best Local LLM by GPU (hub)
- Best Local LLM by RAM (hub)
- The RTX 50 SUPER Is Not Coming — why no cheaper high-VRAM NVIDIA card is arriving this year
Need OpenClaw fixed live?
Remote rescue sessions for gateway, auth, tunnel, VPS, and model access problems.
See Rescue Session