# NVLink and InfiniBand: how AI supercomputers connect thousands of GPUs Training a frontier AI model isn't done on one graphics card — it's done on thousands, working in lockstep. But a GPU is only as useful as its ability to *talk* to the other GPUs, and ordinary connections are far too slow for the torrent of data they must share on every training step. Two technologies solve this at two different scales: **NVLink**, which wires GPUs together inside a server or rack, and **InfiniBand**, which links those racks across an entire datacenter. This guide explains what each one is, how fast they are, how they combine, and why the interconnect — not the GPU alone — often decides how quickly a large model trains. ## Why one GPU is never enough Modern models are too big for a single GPU's memory, and even when they fit, one GPU would take years to train them. So the work is split across many GPUs using **parallelism**: the data, the model's layers, or individual tensors are divided up, each GPU does its share, and then — crucially — the GPUs must *synchronise*. After every step, they exchange and combine their results (a collective operation called an **AllReduce**) so every GPU ends up with the same updated weights.[10](#ref-10) That synchronisation is relentless: gigabytes of gradients, many times per second, across every GPU in the job. If the link between GPUs is slow, they spend their time *waiting* instead of computing, and expensive silicon sits idle. This is why interconnect bandwidth and latency are first-class concerns in AI infrastructure — and why the standard PCIe slot that connects a GPU to the motherboard, at roughly 64 GB/s for a PCIe 5.0 ×16 link, quickly becomes the bottleneck.[8](#ref-8) ## Two scales: scale-up and scale-out The industry splits the problem into two layers, and it's the key mental model for everything below: - **Scale-up** — tightly connecting a handful of GPUs *inside a single server or rack* so they behave almost like one enormous GPU. This is **NVLink**'s job. - **Scale-out** — connecting many such servers *across the datacenter* into a cluster of hundreds or thousands of GPUs. This is **InfiniBand**'s job (Ethernet is the main alternative). ## NVLink: the GPU-to-GPU highway **NVLink** is NVIDIA's direct GPU-to-GPU interconnect. Instead of forcing GPUs to talk through the slower, shared PCIe bus, NVLink gives them dedicated high-speed lanes so they can read and write each other's memory at enormous bandwidth. It was first announced in 2014 and shipped in 2016 with the Pascal-generation Tesla P100, which offered four links for about 160 GB/s of bidirectional bandwidth — already several times what PCIe could do at the time.[3](#ref-3)[4](#ref-4) It has scaled aggressively since. The fifth-generation NVLink used in NVIDIA's Blackwell GPUs delivers **1.8 TB/s of bandwidth per GPU** — 18 links at 100 GB/s each — which is on the order of 14× the bandwidth of a PCIe 5.0 ×16 connection.[1](#ref-1) ### NVSwitch: from a few GPUs to a rack that acts like one Point-to-point links only go so far; to connect many GPUs so that *every* GPU can reach every other at full speed, NVIDIA adds a switch chip called **NVSwitch**. It creates a non-blocking fabric where all GPUs share their combined bandwidth. The flagship example is the **GB200 NVL72**, which uses NVLink and NVSwitch to join 72 Blackwell GPUs into a single rack-scale unit with **130 TB/s of all-to-all bandwidth** — presented to software as one gigantic GPU.[2](#ref-2) ## InfiniBand: the datacenter fabric NVLink connects GPUs within a rack, but frontier training spans *many* racks. Linking them needs a network that is fast, extremely low-latency, and lossless at scale. That is **InfiniBand** — a networking standard governed by the InfiniBand Trade Association (IBTA), formed in 1999, whose defining feature has always been **RDMA** (Remote Direct Memory Access): the ability for one machine to read or write another machine's memory directly, without interrupting its CPU or copying through the operating system.[6](#ref-6) That's what keeps latency low and throughput high across a big cluster. NVIDIA's current platform, **Quantum-2**, runs InfiniBand at the **NDR** data rate of **400 Gb/s per port**. A single Quantum-2 QM9700 switch packs 64 such ports into a 1U chassis for 51.2 Tb/s of aggregate throughput, and the platform adds acceleration engines like **SHARP** (Scalable Hierarchical Aggregation and Reduction Protocol), which performs parts of the AllReduce *inside the network switches* themselves to cut the data that must travel.[5](#ref-5) The next data rate, XDR, doubles per-port speed to 800 Gb/s.[5](#ref-5) ### GPUDirect RDMA: skipping the CPU entirely For AI, InfiniBand pairs with **GPUDirect RDMA**, which lets the network adapter move data straight into and out of a GPU's memory over the PCIe peer-to-peer path — bypassing the CPU and host memory completely.[7](#ref-7) So when 1,024 GPUs across dozens of servers perform an AllReduce, a tensor can travel from one GPU's memory to another's across the datacenter without ever being copied into a CPU's RAM. That end-to-end, GPU-to-GPU path is what makes cluster-scale training practical. ## How they work together Picture a real AI supercomputer. Inside each server (or rack), **NVLink + NVSwitch** bind the GPUs into one tightly-coupled unit. Between servers, **InfiniBand** (with GPUDirect RDMA) carries traffic across the datacenter. NVIDIA's reference cluster designs, such as the DGX SuperPOD, are built exactly this way — NVLink for scale-up within nodes, InfiniBand for scale-out between them — scaling to thousands of GPUs acting as a single system.[9](#ref-9) The software that orchestrates the collective operations, NVIDIA's **NCCL** library, is topology-aware: it automatically routes each part of an AllReduce over the fastest available path — NVLink when two GPUs are in the same node, InfiniBand when they're in different ones.[10](#ref-10) To the model author, it simply looks like “the GPUs stay in sync.” ## NVLink vs. InfiniBand vs. PCIe at a glance | Interconnect | Scope | Role | Example bandwidth | | --- | --- | --- | --- | | PCIe 5.0 | GPU ↔ motherboard/CPU | General I/O; the baseline | ~64 GB/s (×16, one way)[8](#ref-8) | | NVLink 5 + NVSwitch | GPU ↔ GPU, in a server/rack | Scale-up (tight coupling) | 1.8 TB/s per GPU[1](#ref-1) | | InfiniBand NDR (Quantum-2) | Server ↔ server, datacenter | Scale-out (clustering) | 400 Gb/s per port[5](#ref-5) | ## Why this decides how fast models train As GPU counts climb into the thousands, the fraction of each step spent on *communication* rather than *computation* grows. A cluster with slow interconnects hits a wall: adding more GPUs stops making training meaningfully faster, because the extra chips spend their time waiting to synchronise. Fast interconnects push that wall much further out — which is precisely why NVLink bandwidth and InfiniBand data rates get headline billing every GPU generation, right alongside the raw compute numbers. ## Takeaways - **NVLink = scale-up.** Direct GPU-to-GPU links inside a server or rack; NVLink 5 reaches 1.8 TB/s per GPU, and NVSwitch turns dozens of GPUs into one.[1](#ref-1)[2](#ref-2) - **InfiniBand = scale-out.** A low-latency, RDMA-based datacenter fabric; Quantum-2 runs at 400 Gb/s NDR per port with in-network computing.[5](#ref-5) - **RDMA and GPUDirect** let data move GPU-to-GPU across the network without touching a CPU — the trick that makes cluster training efficient.[7](#ref-7) - **They stack.** NVLink inside the node, InfiniBand between nodes, NCCL routing the traffic — together they let thousands of GPUs behave like one machine.[9](#ref-9) - **The interconnect is a bottleneck, not a detail.** Beyond a point, communication speed — not GPU count — sets how fast a large model trains. ### References 1. [NVIDIA — NVLink & NVLink Switch (fifth-generation NVLink: 1.8 TB/s per GPU; NVSwitch fabric)](https://www.nvidia.com/en-us/data-center/nvlink/) 2. [NVIDIA — GB200 NVL72 (72 Blackwell GPUs joined by NVLink/NVSwitch; 130 TB/s all-to-all bandwidth)](https://www.nvidia.com/en-us/data-center/gb200-nvl72/) 3. [NVLink — overview and generation history (announced 2014; debuted with Tesla P100 in 2016)](https://en.wikipedia.org/wiki/NVLink) 4. [NVIDIA Technical Blog — "Inside Pascal": Tesla P100, the first NVLink product (~160 GB/s across four links)](https://developer.nvidia.com/blog/inside-pascal/) 5. [NVIDIA — Quantum-2 InfiniBand Platform (NDR 400 Gb/s per port; QM9700 switch; RDMA and SHARP in-network computing)](https://www.nvidia.com/en-us/networking/quantum2/) 6. [InfiniBand Trade Association — history of InfiniBand and RDMA (IBTA formed 1999; RDMA; the RoCE initiative)](https://www.infinibandta.org/celebrating-25-years-of-the-infiniband-trade-association/) 7. [NVIDIA — GPUDirect RDMA documentation (direct GPU-memory transfers over the network, bypassing the CPU)](https://docs.nvidia.com/cuda/gpudirect-rdma/) 8. [PCI Express — data rates by generation (PCIe 5.0: 32 GT/s per lane; ~64 GB/s on an ×16 link, one direction)](https://en.wikipedia.org/wiki/PCI_Express) 9. [NVIDIA — DGX SuperPOD (reference AI cluster: NVLink for scale-up within nodes, InfiniBand for scale-out between them)](https://www.nvidia.com/en-us/data-center/dgx-superpod/) 10. [NVIDIA — NCCL (Collective Communications Library): AllReduce and other collectives, topology-aware over NVLink and InfiniBand](https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/index.html) --- *Source: [ocxly.com/nvlink-infiniband-explained.html](https://ocxly.com/nvlink-infiniband-explained.html) — OCXLY, free 100% client-side privacy-first tools. NVLink and InfiniBand explained: the two interconnects that let AI supercomputers act like one giant GPU. How each works, their bandwidth and history (NVLink 5 at 1.8 TB/s, InfiniBand NDR at 400 Gb/s), how they combine in a cluster, and why they decide training speed. Cited to primary sources.*