How much VRAM do AI models need? A practical guide

Ask “can my computer run this AI model?” and the answer almost always comes down to one number: VRAM, the memory on your graphics card. Too little and the model simply won't load; enough and it runs. This guide explains what VRAM is, why AI models are so hungry for it, how to estimate a model's appetite from two facts you can look up in seconds, and roughly how much real models — from a 7-billion-parameter chatbot to Llama 405B and the image generators behind today's art tools — actually demand.

What VRAM actually is

VRAM (video RAM) is the dedicated memory that lives on the graphics card itself, right next to the GPU. It is physically separate from your computer's main system RAM. When a program wants the GPU to compute something, the data has to be sitting in VRAM first — the GPU's thousands of cores can't reach into system RAM at usable speed.

Two things make VRAM special. The first is capacity: how many gigabytes it holds. The second, and just as important for AI, is bandwidth: how fast data moves between the memory and the GPU. Consumer cards use a memory type called GDDR — NVIDIA's GeForce RTX 5090, for instance, ships with 32 GB of GDDR7 running at about 1,792 GB/s.¹ Data-center accelerators go a step further and use HBM (high-bandwidth memory), which stacks memory chips vertically and connects them with an extremely wide bus to reach far higher bandwidth than GDDR — which is exactly why HBM is standard on the GPUs used to train large models.²

Why AI models live in VRAM

A neural network is, at heart, a very large pile of numbers called parameters (or weights). To run the model, every one of those numbers has to be held in memory the GPU can read at speed. On top of the weights, running a model needs room for three more things:

• Activations — the intermediate results computed as data flows through the network.
• The KV cache — for language models, a running store of past context that grows with how much text you feed in (more on this below).
• Overhead — temporary buffers and workspace the framework needs.

Because modern models are dominated by huge matrix multiplications that stream weights through the cores, they tend to be memory-bandwidth-bound: the GPU often finishes the arithmetic faster than memory can feed it. That is why both the size and the speed of VRAM matter, and why “just add more system RAM” doesn't help.

The one formula that matters

Here's the trick that lets you estimate almost any model's memory in your head. A model's weight memory is simply its parameter count × the number of bytes used to store each parameter. That second factor is the model's precision, and it's the single biggest lever you control:

So the rule of thumb is: a model with N billion parameters needs about N×2 GB of VRAM at FP16, or N×0.5 GB at 4-bit, just for the weights. Hugging Face's own memory estimator uses exactly this arithmetic — multiply parameters by bytes-per-parameter — and then adds up to about 20% on top for the activations and workspace that inference needs.³

weight memory (GB)  =  params (billions) x bytes_per_param
inference VRAM      ≈  weight memory x 1.2      # +~20% overhead

# Example: a 7B model at FP16
#   7 x 2 = 14 GB weights  ->  ~17 GB to actually run
# The same model at 4-bit
#   7 x 0.5 = 3.5 GB       ->  ~4-5 GB to run

Inference vs. training: a 10× difference

Everything above is for inference — running a finished model. Training (or full fine-tuning) is a different universe. To update weights, the GPU must also hold gradients and the optimizer's bookkeeping. For the standard mixed-precision setup with the AdamW optimizer, Hugging Face breaks the cost down to roughly 18 bytes per parameter: about 6 for two copies of the weights, 8 for Adam's momentum and variance, and 4 for the gradients — and that's before activations.⁴

Concretely, fully fine-tuning a 7B model can require on the order of 7 × 18 ≈ 126 GB — far past any single consumer GPU. This memory blow-up is precisely what techniques like ZeRO were designed to attack, by splitting the optimizer states, gradients, and weights across many GPUs instead of duplicating them.⁷ It is also why almost nobody full-fine-tunes at home; they use parameter-efficient methods instead (see quantization, below).

The hidden cost: the KV cache and context length

Language models based on the Transformer use attention, which lets each new token look back at every previous token.⁵ To avoid recomputing the past on every step, the model caches the “keys” and “values” for all previous tokens — the KV cache. The catch: this cache grows with the number of tokens in play, so a long conversation or a 100,000-token document can consume many gigabytes on its own, on top of the weights. Serving many users at once multiplies it further. Managing this memory efficiently is such a hard problem that it spawned dedicated systems like vLLM's PagedAttention, which treats the KV cache like virtual memory to stop it from wasting VRAM.⁶

Quantization: how models got small enough to run at home

Quantization stores each weight in fewer bits — 8-bit or even 4-bit instead of 16 — shrinking the model with surprisingly little quality loss. It is the reason a laptop can run models that once needed a server. The landmark result is QLoRA, which introduced a 4-bit data type (NF4) and paged optimizers to fine-tune a 65-billion-parameter model on a single 48 GB GPU while keeping the quality of full 16-bit fine-tuning.⁸ For plain inference, 4-bit roughly quarters the FP16 footprint: that 7B model drops from ~14 GB to ~3.5 GB.³

How much VRAM real models need

Putting it together, here are ballpark figures for popular models. These are approximate weights-only numbers from the parameter-count-×-precision rule; add headroom for context and overhead as discussed.

Large language models

The jump is stark. An 8B model is comfortable on a mid-range gaming card; the 70B needs a high-end 48 GB workstation card (or two 24 GB cards) even at 4-bit; and Llama 405B — a model Meta trained on a cluster of more than 16,000 H100 GPUs — needs a multi-GPU server no matter how aggressively you quantize.⁹

Image, video, and speech models

Note how much heavier FLUX.1 is than the Stable Diffusion family: with about 12 billion parameters in its diffusion transformer, it needs roughly 24 GB at half precision — though FP8 halves that to about 12 GB, and 4-bit builds squeeze it onto 8 GB cards.¹³ Whisper's speech models, by contrast, span a wide range: the family runs from a 39-million-parameter “tiny” up to the 1.5-billion-parameter “large,” so you can pick a size to fit almost any card.¹⁴

Matching models to graphics cards

To translate all of this into a shopping decision, here's where common GPUs land. Capacity is what determines whether a model fits; bandwidth largely determines how fast it runs once it does.

Rules of thumb to remember

• Weights ≈ parameters × bytes-per-parameter. FP16 = 2 bytes, 4-bit = 0.5 bytes. A 13B model is ~26 GB at FP16, ~6.5 GB at 4-bit.³
• Add ~20% for inference, and much more for long context or large batches.³
• Training costs ~10× inference because of gradients and optimizer states (~18 bytes/param).⁴
• Quantize to fit. 4-bit is the great equalizer — it's what puts capable models on ordinary hardware.⁸
• Capacity decides if it runs; bandwidth decides how fast. Both are properties of VRAM, and neither can be substituted with system RAM.²