# How much VRAM do AI models need? A practical guide Ask “can my computer run this AI model?” and the answer almost always comes down to one number: **VRAM**, the memory on your graphics card. Too little and the model simply won't load; enough and it runs. This guide explains what VRAM is, why AI models are so hungry for it, how to estimate a model's appetite from two facts you can look up in seconds, and roughly how much real models — from a 7-billion-parameter chatbot to Llama 405B and the image generators behind today's art tools — actually demand. ## What VRAM actually is VRAM (video RAM) is the dedicated memory that lives *on the graphics card itself*, right next to the GPU. It is physically separate from your computer's main system RAM. When a program wants the GPU to compute something, the data has to be sitting in VRAM first — the GPU's thousands of cores can't reach into system RAM at usable speed. Two things make VRAM special. The first is **capacity**: how many gigabytes it holds. The second, and just as important for AI, is **bandwidth**: how fast data moves between the memory and the GPU. Consumer cards use a memory type called GDDR — NVIDIA's GeForce RTX 5090, for instance, ships with 32 GB of GDDR7 running at about 1,792 GB/s.[1](#ref-1) Data-center accelerators go a step further and use **HBM** (high-bandwidth memory), which stacks memory chips vertically and connects them with an extremely wide bus to reach far higher bandwidth than GDDR — which is exactly why HBM is standard on the GPUs used to train large models.[2](#ref-2) ## Why AI models live in VRAM A neural network is, at heart, a very large pile of numbers called **parameters** (or weights). To run the model, every one of those numbers has to be held in memory the GPU can read at speed. On top of the weights, running a model needs room for three more things: - **Activations** — the intermediate results computed as data flows through the network. - **The KV cache** — for language models, a running store of past context that grows with how much text you feed in (more on this below). - **Overhead** — temporary buffers and workspace the framework needs. Because modern models are dominated by huge matrix multiplications that stream weights through the cores, they tend to be *memory-bandwidth-bound*: the GPU often finishes the arithmetic faster than memory can feed it. That is why both the *size* and the *speed* of VRAM matter, and why “just add more system RAM” doesn't help. ## The one formula that matters Here's the trick that lets you estimate almost any model's memory in your head. A model's weight memory is simply its **parameter count × the number of bytes used to store each parameter**. That second factor is the model's *precision*, and it's the single biggest lever you control: | Precision | Bytes / parameter | Memory for a 7B model (weights) | | --- | --- | --- | | FP32 (full) | 4 | ~28 GB | | FP16 / BF16 (half) | 2 | ~14 GB | | INT8 / FP8 (8-bit) | 1 | ~7 GB | | INT4 (4-bit) | 0.5 | ~3.5 GB | So the rule of thumb is: **a model with *N* billion parameters needs about *N*×2 GB of VRAM at FP16, or *N*×0.5 GB at 4-bit**, just for the weights. Hugging Face's own memory estimator uses exactly this arithmetic — multiply parameters by bytes-per-parameter — and then adds up to about 20% on top for the activations and workspace that inference needs.[3](#ref-3) ``` weight memory (GB) = params (billions) x bytes_per_param inference VRAM ≈ weight memory x 1.2 # +~20% overhead # Example: a 7B model at FP16 # 7 x 2 = 14 GB weights -> ~17 GB to actually run # The same model at 4-bit # 7 x 0.5 = 3.5 GB -> ~4-5 GB to run ``` ## Inference vs. training: a 10× difference Everything above is for **inference** — running a finished model. **Training** (or full fine-tuning) is a different universe. To update weights, the GPU must also hold gradients and the optimizer's bookkeeping. For the standard mixed-precision setup with the AdamW optimizer, Hugging Face breaks the cost down to roughly **18 bytes per parameter**: about 6 for two copies of the weights, 8 for Adam's momentum and variance, and 4 for the gradients — and that's *before* activations.[4](#ref-4) Concretely, fully fine-tuning a 7B model can require on the order of 7 × 18 ≈ 126 GB — far past any single consumer GPU. This memory blow-up is precisely what techniques like ZeRO were designed to attack, by splitting the optimizer states, gradients, and weights across many GPUs instead of duplicating them.[7](#ref-7) It is also why almost nobody full-fine-tunes at home; they use parameter-efficient methods instead (see quantization, below). ## The hidden cost: the KV cache and context length Language models based on the Transformer use **attention**, which lets each new token look back at every previous token.[5](#ref-5) To avoid recomputing the past on every step, the model caches the “keys” and “values” for all previous tokens — the **KV cache**. The catch: this cache grows with the number of tokens in play, so a long conversation or a 100,000-token document can consume many gigabytes on its own, on top of the weights. Serving many users at once multiplies it further. Managing this memory efficiently is such a hard problem that it spawned dedicated systems like vLLM's PagedAttention, which treats the KV cache like virtual memory to stop it from wasting VRAM.[6](#ref-6) ## Quantization: how models got small enough to run at home **Quantization** stores each weight in fewer bits — 8-bit or even 4-bit instead of 16 — shrinking the model with surprisingly little quality loss. It is the reason a laptop can run models that once needed a server. The landmark result is **QLoRA**, which introduced a 4-bit data type (NF4) and paged optimizers to fine-tune a *65-billion-parameter* model on a single 48 GB GPU while keeping the quality of full 16-bit fine-tuning.[8](#ref-8) For plain inference, 4-bit roughly quarters the FP16 footprint: that 7B model drops from ~14 GB to ~3.5 GB.[3](#ref-3) ## How much VRAM real models need Putting it together, here are ballpark figures for popular models. These are approximate *weights-only* numbers from the parameter-count-×-precision rule; add headroom for context and overhead as discussed. ### Large language models | Model | Parameters | FP16 | 4-bit | | --- | --- | --- | --- | | Mistral 7B[10](#ref-10) | 7B | ~14 GB | ~4 GB | | Llama 3.1 8B[9](#ref-9) | 8B | ~16 GB | ~5 GB | | Llama 3.1 70B[9](#ref-9) | 70B | ~140 GB | ~40 GB | | Llama 3.1 405B[9](#ref-9) | 405B | ~810 GB | ~230 GB | The jump is stark. An 8B model is comfortable on a mid-range gaming card; the 70B needs a high-end 48 GB workstation card (or two 24 GB cards) even at 4-bit; and Llama 405B — a model Meta trained on a cluster of more than 16,000 H100 GPUs — needs a multi-GPU server no matter how aggressively you quantize.[9](#ref-9) ### Image, video, and speech models | Model | Size | Typical VRAM to run | | --- | --- | --- | | Stable Diffusion 1.5[12](#ref-12) | ~0.9B UNet | 4 GB min, 6–8 GB comfortable | | SDXL[11](#ref-11) | 2.6B UNet | 8 GB min, 12 GB recommended | | FLUX.1 [dev][13](#ref-13) | 12B | ~24 GB (FP16); ~12 GB (FP8); 6–8 GB (4-bit) | | Whisper large[14](#ref-14) | 1.5B | ~3–5 GB | Note how much heavier FLUX.1 is than the Stable Diffusion family: with about 12 billion parameters in its diffusion transformer, it needs roughly 24 GB at half precision — though FP8 halves that to about 12 GB, and 4-bit builds squeeze it onto 8 GB cards.[13](#ref-13) Whisper's speech models, by contrast, span a wide range: the family runs from a 39-million-parameter “tiny” up to the 1.5-billion-parameter “large,” so you can pick a size to fit almost any card.[14](#ref-14) ## Matching models to graphics cards To translate all of this into a shopping decision, here's where common GPUs land. Capacity is what determines whether a model *fits*; bandwidth largely determines how *fast* it runs once it does. | GPU | VRAM | Memory type | Comfortable with | | --- | --- | --- | --- | | RTX 3060 | 12 GB | GDDR6 | 7–8B LLMs (4-bit), SD 1.5, SDXL | | RTX 4090 | 24 GB | GDDR6X | 8B LLMs (FP16), ~30B (4-bit), FLUX.1 | | RTX 5090[1](#ref-1) | 32 GB | GDDR7 | Larger 4-bit models, headroom for context | | A100 / H100[15](#ref-15) | 80 GB | HBM2e / HBM3 | 70B-class models; the workhorses of training | ## Rules of thumb to remember - **Weights ≈ parameters × bytes-per-parameter.** FP16 = 2 bytes, 4-bit = 0.5 bytes. A 13B model is ~26 GB at FP16, ~6.5 GB at 4-bit.[3](#ref-3) - **Add ~20% for inference**, and much more for long context or large batches.[3](#ref-3) - **Training costs ~10× inference** because of gradients and optimizer states (~18 bytes/param).[4](#ref-4) - **Quantize to fit.** 4-bit is the great equalizer — it's what puts capable models on ordinary hardware.[8](#ref-8) - **Capacity decides *if* it runs; bandwidth decides *how fast*.** Both are properties of VRAM, and neither can be substituted with system RAM.[2](#ref-2) ### References 1. [NVIDIA — GeForce RTX 5090 (32 GB GDDR7; memory bandwidth and specifications)](https://www.nvidia.com/en-us/geforce/graphics-cards/50-series/rtx-5090/) 2. [High Bandwidth Memory (HBM) — 3D-stacked DRAM and wide-bus bandwidth vs. GDDR (encyclopedic overview)](https://en.wikipedia.org/wiki/High_Bandwidth_Memory) 3. [Hugging Face — Model memory estimator: bytes per parameter by precision and the ~20% inference overhead](https://huggingface.co/docs/accelerate/usage_guides/model_size_estimator) 4. [Hugging Face — Model memory anatomy: ~18 bytes/parameter for mixed-precision AdamW training (weights, optimizer states, gradients)](https://huggingface.co/docs/transformers/en/model_memory_anatomy) 5. [Vaswani et al. (2017) — "Attention Is All You Need" (the Transformer and attention)](https://arxiv.org/abs/1706.03762) 6. [Kwon et al. (2023) — "Efficient Memory Management for Large Language Model Serving with PagedAttention" (vLLM; KV-cache memory)](https://arxiv.org/abs/2309.06180) 7. [Rajbhandari et al. (2020) — "ZeRO: Memory Optimizations Toward Training Trillion Parameter Models"](https://arxiv.org/abs/1910.02054) 8. [Dettmers et al. (2023) — "QLoRA: Efficient Finetuning of Quantized LLMs" (4-bit NF4; 65B on a single 48 GB GPU)](https://arxiv.org/abs/2305.14314) 9. [Meta / Hugging Face — "Llama 3.1: 405B, 70B & 8B" (parameter sizes, 128K context, training scale)](https://huggingface.co/blog/llama31) 10. [Jiang et al. (2023) — "Mistral 7B" (7-billion-parameter model)](https://arxiv.org/abs/2310.06825) 11. [Podell et al. (2023) — "SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis" (2.6B UNet)](https://arxiv.org/abs/2307.01952) 12. [Rombach et al. (2022) — "High-Resolution Image Synthesis with Latent Diffusion Models" (the basis of Stable Diffusion)](https://arxiv.org/abs/2112.10752) 13. [Black Forest Labs — FLUX.1 [dev] model card (~12B-parameter diffusion transformer; VRAM by precision)](https://huggingface.co/black-forest-labs/FLUX.1-dev) 14. [Radford et al. (2022) — "Robust Speech Recognition via Large-Scale Weak Supervision" (Whisper; model sizes 39M–1.5B)](https://arxiv.org/abs/2212.04356) 15. [NVIDIA — H100 Tensor Core GPU (80 GB HBM; data-center training accelerator)](https://www.nvidia.com/en-us/data-center/h100/) --- *Source: [ocxly.com/vram-requirements-ai-models.html](https://ocxly.com/vram-requirements-ai-models.html) — OCXLY, free 100% client-side privacy-first tools. What VRAM is, why AI models need it, and how to estimate GPU memory from a model's parameter count and precision. Includes concrete VRAM figures for Llama 3.1, Mistral, Stable Diffusion, SDXL, FLUX.1 and Whisper, plus a GPU buying guide. Cited to primary sources.*