How much VRAM do AI models need? A practical guide

Ask “can my computer run this AI model?” and the answer almost always comes down to one number: VRAM, the memory on your graphics card. Too little and the model simply won't load; enough and it runs. This guide explains what VRAM is, why AI models are so hungry for it, how to estimate a model's appetite from two facts you can look up in seconds, and roughly how much real models — from a 7-billion-parameter chatbot to Llama 405B and the image generators behind today's art tools — actually demand.

What VRAM actually is

VRAM (video RAM) is the dedicated memory that lives on the graphics card itself, right next to the GPU. It is physically separate from your computer's main system RAM. When a program wants the GPU to compute something, the data has to be sitting in VRAM first — the GPU's thousands of cores can't reach into system RAM at usable speed.

Two things make VRAM special. The first is capacity: how many gigabytes it holds. The second, and just as important for AI, is bandwidth: how fast data moves between the memory and the GPU. Consumer cards use a memory type called GDDR — NVIDIA's GeForce RTX 5090, for instance, ships with 32 GB of GDDR7 running at about 1,792 GB/s.¹ Data-center accelerators go a step further and use HBM (high-bandwidth memory), which stacks memory chips vertically and connects them with an extremely wide bus to reach far higher bandwidth than GDDR — which is exactly why HBM is standard on the GPUs used to train large models.²

Why AI models live in VRAM

A neural network is, at heart, a very large pile of numbers called parameters (or weights). To run the model, every one of those numbers has to be held in memory the GPU can read at speed. On top of the weights, running a model needs room for three more things:

• Activations — the intermediate results computed as data flows through the network.
• The KV cache — for language models, a running store of past context that grows with how much text you feed in (more on this below).
• Overhead — temporary buffers and workspace the framework needs.

Because modern models are dominated by huge matrix multiplications that stream weights through the cores, they tend to be memory-bandwidth-bound: the GPU often finishes the arithmetic faster than memory can feed it. That is why both the size and the speed of VRAM matter, and why “just add more system RAM” doesn't help.

The one formula that matters

Here's the trick that lets you estimate almost any model's memory in your head. A model's weight memory is simply its parameter count × the number of bytes used to store each parameter. That second factor is the model's precision, and it's the single biggest lever you control:

So the rule of thumb is: a model with N billion parameters needs about N×2 GB of VRAM at FP16, or N×0.5 GB at 4-bit, just for the weights. Hugging Face's own memory estimator uses exactly this arithmetic — multiply parameters by bytes-per-parameter — and then adds up to about 20% on top for the activations and workspace that inference needs.³

weight memory (GB)  =  params (billions) x bytes_per_param
inference VRAM      ≈  weight memory x 1.2      # +~20% overhead

# Example: a 7B model at FP16
#   7 x 2 = 14 GB weights  ->  ~17 GB to actually run
# The same model at 4-bit
#   7 x 0.5 = 3.5 GB       ->  ~4-5 GB to run

Mixture-of-Experts: when the parameter count misleads

There is one big asterisk on the formula above. A growing share of frontier models are Mixture-of-Experts (MoE): instead of one dense network, the model is split into many “expert” sub-networks, and a small router sends each token to just a few of them. That decouples two numbers people usually treat as one — total parameters and active parameters. DeepSeek-V3, for instance, has 671 billion total parameters but activates only about 37 billion per token.¹⁶

The trap for VRAM budgeting: the memory formula uses the total parameter count, not the active one. Every expert has to be resident in memory in case the router picks it, so a 671B MoE needs room for all 671B weights — on the order of 340 GB even at 4-bit — despite each token only touching 37B of them. What MoE buys is compute and bandwidth per token: inference runs about as fast as a dense ~37B model. What it does not buy is capacity. MoE models are cheap to run but expensive to hold, which is exactly why they are pushing interest toward large-capacity, lower-bandwidth hardware (below).

Inference vs. training: a 10× difference

Everything above is for inference — running a finished model. Training (or full fine-tuning) is a different universe. To update weights, the GPU must also hold gradients and the optimizer's bookkeeping. For the standard mixed-precision setup with the AdamW optimizer, Hugging Face breaks the cost down to roughly 18 bytes per parameter: about 6 for two copies of the weights, 8 for Adam's momentum and variance, and 4 for the gradients — and that's before activations.⁴

Concretely, fully fine-tuning a 7B model can require on the order of 7 × 18 ≈ 126 GB — far past any single consumer GPU. This memory blow-up is precisely what techniques like ZeRO were designed to attack, by splitting the optimizer states, gradients, and weights across many GPUs instead of duplicating them.⁷ It is also why almost nobody full-fine-tunes at home; they use parameter-efficient methods instead (see quantization, below).

The hidden cost: the KV cache and context length

Language models based on the Transformer use attention, which lets each new token look back at every previous token.⁵ To avoid recomputing the past on every step, the model caches the “keys” and “values” for all previous tokens — the KV cache. The catch: this cache grows with the number of tokens in play, so a long conversation or a 100,000-token document can consume many gigabytes on its own, on top of the weights. Serving many users at once multiplies it further. Managing this memory efficiently is such a hard problem that it spawned dedicated systems like vLLM's PagedAttention, which treats the KV cache like virtual memory to stop it from wasting VRAM.⁶

Quantization: how models got small enough to run at home

Quantization stores each weight in fewer bits — 8-bit or even 4-bit instead of 16 — shrinking the model with surprisingly little quality loss. It is the reason a laptop can run models that once needed a server. The landmark result is QLoRA, which introduced a 4-bit data type (NF4) and paged optimizers to fine-tune a 65-billion-parameter model on a single 48 GB GPU while keeping the quality of full 16-bit fine-tuning.⁸ For plain inference, 4-bit roughly quarters the FP16 footprint: that 7B model drops from ~14 GB to ~3.5 GB.³

The newest twist is hardware-native 4-bit floating-point. NVIDIA's Blackwell GPUs add an FP4/NVFP4 format that turns 4-bit from a storage trick the CPU has to unpack into a first-class compute format the tensor cores run directly — the same ~0.5 bytes per parameter, but faster, and increasingly the default for shipping quantized models on current hardware.¹⁸

Running a model bigger than your card

When a model still won't fit even after quantization, the last resort is offloading: keep part of it in system RAM (or even on disk) and stream pieces to the GPU as they are needed. Tools like llama.cpp let you place a chosen number of a model's layers on the GPU and run the remainder on the CPU, so a 70B model can technically run on an 8 GB card — it just splits the work across whatever memory you have.¹⁹

The price is speed, and the drop is steep. Every weight that lives outside VRAM has to cross the comparatively slow PCIe link on each use, so generation can fall from dozens of tokens per second to a handful, sometimes less than one. Offloading is what turns an “impossible” model into a merely slow one — and it is precisely why a big-but-slow, large-memory machine can beat a fast-but-small discrete GPU for models that would otherwise not load at all.

How much VRAM real models need

Putting it together, here are ballpark figures for popular models. These are approximate weights-only numbers from the parameter-count-×-precision rule; add headroom for context and overhead as discussed.

Large language models

The jump is stark. An 8B model is comfortable on a mid-range gaming card; the 70B needs a high-end 48 GB workstation card (or two 24 GB cards) even at 4-bit; and Llama 405B — a model Meta trained on a cluster of more than 16,000 H100 GPUs — needs a multi-GPU server no matter how aggressively you quantize.⁹

Image, video, and speech models

Note how much heavier FLUX.1 is than the Stable Diffusion family: with about 12 billion parameters in its diffusion transformer, it needs roughly 24 GB at half precision — though FP8 halves that to about 12 GB, and 4-bit builds squeeze it onto 8 GB cards.¹³ Whisper's speech models, by contrast, span a wide range: the family runs from a 39-million-parameter “tiny” up to the 1.5-billion-parameter “large,” so you can pick a size to fit almost any card.¹⁴

Matching models to graphics cards

To translate all of this into a shopping decision, here's where common GPUs land. Capacity is what determines whether a model fits; bandwidth largely determines how fast it runs once it does.

Unified memory: when system RAM becomes VRAM

The strict VRAM-versus-system-RAM split this guide opened with is being blurred by a new class of chips built around unified memory, where the CPU and GPU share a single physical pool instead of each keeping its own. Apple's M-series Macs pioneered this at scale — a Mac Studio can be configured with up to 512 GB of unified memory the GPU addresses directly. AMD's Ryzen AI Max+ 395 (“Strix Halo”) carries up to 128 GB of unified LPDDR5X, as much as 96 GB of which can be handed to the GPU — enough to hold a 70B model at 4-bit inside a laptop-class chip.¹⁷ NVIDIA's DGX Spark wraps 128 GB of coherent unified memory around a GB10 Grace Blackwell chip aimed squarely at local AI.¹⁸

The catch is bandwidth. Unified LPDDR5X runs at a few hundred GB/s — the Ryzen AI Max tops out around 250 GB/s — against roughly 1,800 GB/s on an RTX 5090 and several terabytes per second on HBM. So these machines win decisively on capacity, fitting models a 24 GB card never could, while generating tokens more slowly once the model is loaded. They are the clearest illustration of this guide's core rule, now playing out at the level of whole computers: capacity decides whether it runs; bandwidth decides how fast.

Rules of thumb to remember

• Weights ≈ parameters × bytes-per-parameter. FP16 = 2 bytes, 4-bit = 0.5 bytes. A 13B model is ~26 GB at FP16, ~6.5 GB at 4-bit.³
• Add ~20% for inference, and much more for long context or large batches.³
• Training costs ~10× inference because of gradients and optimizer states (~18 bytes/param).⁴
• Quantize to fit. 4-bit is the great equalizer — it's what puts capable models on ordinary hardware.⁸
• Capacity decides if it runs; bandwidth decides how fast. Both are properties of VRAM, and neither can be substituted with system RAM.²