# The 2026 hardware reality check: what it takes to run local LLMs The belief that you need a $3,000 graphics card to run AI at home is out of date. Two shifts — 4-bit quantization and unified memory — quietly moved capable local models onto ordinary laptops. Here is the honest math. Ask a beginner what it takes to run a language model at home and you will usually hear some version of "a very expensive NVIDIA GPU." That was true a couple of years ago. It is not true now, and the reason is two unglamorous engineering shifts — quantization and unified memory — that between them turned "needs a data-centre card" into "runs on the laptop you already own." This guide is the math, without the marketing. ## 01 The one formula that explains everything Almost every hardware question reduces to a single rule of thumb: the memory a model needs is roughly its **parameter count times the bytes used per parameter**, plus some overhead for context. In full 16-bit precision each parameter takes 2 bytes, so a 7-billion-parameter model wants about 14 GB just for weights.[1](#ref-1) That is what scared everyone off. The trick is that you rarely need full precision. ``` # Memory for weights ≈ parameters × bytes-per-parameter (+ context overhead) # FP16 : 2.0 bytes/param -> 7B ≈ 14 GB # 8-bit: 1.0 byte/param -> 7B ≈ 7 GB # 4-bit: 0.5 byte/param -> 7B ≈ 3.5 GB (+ ~1-1.5 GB context ≈ ~5 GB) ``` ## 02 The quantization magic **Quantization** stores each weight at lower precision — most popularly 4-bit integers instead of 16-bit floats. Going from 16-bit to 4-bit is a **4× size reduction (about 75% smaller)**, and the widely-used `Q4_K_M` format from llama.cpp is considered the sweet spot precisely because it delivers that shrink with only minor quality loss.[2](#ref-2) That is the whole game: 4-bit quantization is what lets a 7B model that wanted 14 GB run in roughly 5 GB, and a 14B model — which would need ~28 GB at full precision — sit comfortably inside a 12 GB GPU.[2](#ref-2) You are trading a sliver of accuracy for three-quarters of the memory, and for local use that trade is almost always worth it. ## 03 VRAM vs unified memory: why Apple Silicon changed the value equation A discrete NVIDIA GPU keeps the model in its own dedicated **VRAM**, which is very fast but fixed and expensive — a 24 GB card is a serious purchase, and once the model exceeds that VRAM it simply will not load. Apple Silicon (M3/M4/M5) takes a different approach: **unified memory**, a single pool the CPU, GPU, and Neural Engine all address directly, with no copy across a PCIe bus.[3](#ref-3) Because that pool is the machine's whole RAM, a Mac with 64 or 128 GB of unified memory can load models that no consumer GPU can fit in VRAM at any sane price. The honest trade-off: a top discrete GPU still has far higher memory *bandwidth*, so it generates tokens faster on models that fit in its VRAM.[4](#ref-4) But for the specific goal of running *large* models locally without a data-centre budget, high-memory Apple Silicon is currently the best value going — you buy capacity (fit the model at all) where the GPU buys speed (run a smaller model faster). Pick the axis your workload actually cares about. ## 04 A buyer's guide by memory tier Because it all comes back to memory, you can shop by tier. Assume 4-bit models and leave a couple of gigabytes of headroom for context: - **8 GB (VRAM or unified)** — the entry tier. Runs 7–8B models at 4-bit comfortably (Qwen3 8B, Mistral 7B, Phi-4-mini), which is enough for a genuinely useful local assistant and coding helper.[5](#ref-5) - **16 GB** — the sweet spot. Runs 14B-class models at 4-bit with room to spare, and smaller models fast. This is where most people should aim. - **24 GB** — the enthusiast tier. Comfortably runs ~32B models at 4-bit, the point where local quality starts to feel genuinely strong. - **64–128 GB unified (Apple Silicon)** — the "run the big ones" tier: 70B-class and large Mixture-of-Experts models that no single consumer GPU can hold.[3](#ref-3) > You are not buying teraflops; you are buying gigabytes. The question that decides what you can run locally is almost never "how fast is the chip" — it is "how big is the memory, and how small can I make the model." ## 05 Where OcxlyDev lands Start with the machine you own. A modern laptop with 16 GB can already run a capable 4-bit model, and you should confirm you *like* running models locally before spending anything. If you then want to go bigger, the decision is clean: buy a used 24 GB NVIDIA card if you want maximum speed on models that fit it, or a high-memory Apple Silicon machine if you want to run large models at all. Either way, the $3,000-GPU myth is dead — quantization and unified memory killed it, and the rest of this series is about what to actually *do* with the capable, affordable local models that resulted. **About this piece.** This is part one of a five-part OcxlyDev field guide on running LLMs locally — the hardware reality check, offline RAG and privacy, LM Studio vs Ollama vs Unsloth, the bounded tasks local models win, and the rise of sub-10B models. Every load-bearing claim links to a primary or reputable source below; the local-AI landscape moves monthly, so model names and figures are attributed and directional — check the live page before quoting specifics. ## References 1. [Hugging Face Documentation — "Model training anatomy": how parameter count and precision determine model memory](https://huggingface.co/docs/transformers/en/model_memory_anatomy) 2. [llama.cpp — quantization README: GGUF quant types, ~4× size reduction at 4-bit, and the Q4_K_M sweet spot](https://github.com/ggml-org/llama.cpp/blob/master/tools/quantize/README.md) 3. [Apple — MLX: an array framework for Apple Silicon whose unified-memory model lets CPU and GPU share one pool](https://github.com/ml-explore/mlx) 4. [NVIDIA — GeForce RTX 5090: an example of high-bandwidth dedicated VRAM (fast, but fixed and costly)](https://www.nvidia.com/en-us/geforce/graphics-cards/50-series/rtx-5090/) 5. [Ollama — model library: parameter sizes and quantized download sizes for Qwen3, Mistral, Phi-4 and more](https://ollama.com/library)