NVLink and InfiniBand: how AI supercomputers connect thousands of GPUs
Training a frontier AI model isn't done on one graphics card — it's done on thousands, working in lockstep. But a GPU is only as useful as its ability to talk to the other GPUs, and ordinary connections are far too slow for the torrent of data they must share on every training step. Two technologies solve this at two different scales: NVLink, which wires GPUs together inside a server or rack, and InfiniBand, which links those racks across an entire datacenter. This guide explains what each one is, how fast they are, how they combine, and why the interconnect — not the GPU alone — often decides how quickly a large model trains.
Why one GPU is never enough
Modern models are too big for a single GPU's memory, and even when they fit, one GPU would take years to train them. So the work is split across many GPUs using parallelism: the data, the model's layers, or individual tensors are divided up, each GPU does its share, and then — crucially — the GPUs must synchronise. After every step, they exchange and combine their results (a collective operation called an AllReduce) so every GPU ends up with the same updated weights.10
That synchronisation is relentless: gigabytes of gradients, many times per second, across every GPU in the job. If the link between GPUs is slow, they spend their time waiting instead of computing, and expensive silicon sits idle. This is why interconnect bandwidth and latency are first-class concerns in AI infrastructure — and why the standard PCIe slot that connects a GPU to the motherboard, at roughly 64 GB/s for a PCIe 5.0 ×16 link, quickly becomes the bottleneck.8
Two scales: scale-up and scale-out
The industry splits the problem into two layers, and it's the key mental model for everything below:
- Scale-up — tightly connecting a handful of GPUs inside a single server or rack so they behave almost like one enormous GPU. This is NVLink's job.
- Scale-out — connecting many such servers across the datacenter into a cluster of hundreds or thousands of GPUs. This is InfiniBand's job (Ethernet is the main alternative).
NVLink: the GPU-to-GPU highway
NVLink is NVIDIA's direct GPU-to-GPU interconnect. Instead of forcing GPUs to talk through the slower, shared PCIe bus, NVLink gives them dedicated high-speed lanes so they can read and write each other's memory at enormous bandwidth. It was first announced in 2014 and shipped in 2016 with the Pascal-generation Tesla P100, which offered four links for about 160 GB/s of bidirectional bandwidth — already several times what PCIe could do at the time.34
It has scaled aggressively since. The fifth-generation NVLink used in NVIDIA's Blackwell GPUs delivers 1.8 TB/s of bandwidth per GPU — 18 links at 100 GB/s each — which is on the order of 14× the bandwidth of a PCIe 5.0 ×16 connection.1
NVSwitch: from a few GPUs to a rack that acts like one
Point-to-point links only go so far; to connect many GPUs so that every GPU can reach every other at full speed, NVIDIA adds a switch chip called NVSwitch. It creates a non-blocking fabric where all GPUs share their combined bandwidth. The flagship example is the GB200 NVL72, which uses NVLink and NVSwitch to join 72 Blackwell GPUs into a single rack-scale unit with 130 TB/s of all-to-all bandwidth — presented to software as one gigantic GPU.2
InfiniBand: the datacenter fabric
NVLink connects GPUs within a rack, but frontier training spans many racks. Linking them needs a network that is fast, extremely low-latency, and lossless at scale. That is InfiniBand — a networking standard governed by the InfiniBand Trade Association (IBTA), formed in 1999, whose defining feature has always been RDMA (Remote Direct Memory Access): the ability for one machine to read or write another machine's memory directly, without interrupting its CPU or copying through the operating system.6 That's what keeps latency low and throughput high across a big cluster.
NVIDIA's current platform, Quantum-2, runs InfiniBand at the NDR data rate of 400 Gb/s per port. A single Quantum-2 QM9700 switch packs 64 such ports into a 1U chassis for 51.2 Tb/s of aggregate throughput, and the platform adds acceleration engines like SHARP (Scalable Hierarchical Aggregation and Reduction Protocol), which performs parts of the AllReduce inside the network switches themselves to cut the data that must travel.5 The next data rate, XDR, doubles per-port speed to 800 Gb/s.5
GPUDirect RDMA: skipping the CPU entirely
For AI, InfiniBand pairs with GPUDirect RDMA, which lets the network adapter move data straight into and out of a GPU's memory over the PCIe peer-to-peer path — bypassing the CPU and host memory completely.7 So when 1,024 GPUs across dozens of servers perform an AllReduce, a tensor can travel from one GPU's memory to another's across the datacenter without ever being copied into a CPU's RAM. That end-to-end, GPU-to-GPU path is what makes cluster-scale training practical.
How they work together
Picture a real AI supercomputer. Inside each server (or rack), NVLink + NVSwitch bind the GPUs into one tightly-coupled unit. Between servers, InfiniBand (with GPUDirect RDMA) carries traffic across the datacenter. NVIDIA's reference cluster designs, such as the DGX SuperPOD, are built exactly this way — NVLink for scale-up within nodes, InfiniBand for scale-out between them — scaling to thousands of GPUs acting as a single system.9
The software that orchestrates the collective operations, NVIDIA's NCCL library, is topology-aware: it automatically routes each part of an AllReduce over the fastest available path — NVLink when two GPUs are in the same node, InfiniBand when they're in different ones.10 To the model author, it simply looks like “the GPUs stay in sync.”
NVLink vs. InfiniBand vs. PCIe at a glance
| Interconnect | Scope | Role | Example bandwidth |
|---|---|---|---|
| PCIe 5.0 | GPU ↔ motherboard/CPU | General I/O; the baseline | ~64 GB/s (×16, one way)8 |
| NVLink 5 + NVSwitch | GPU ↔ GPU, in a server/rack | Scale-up (tight coupling) | 1.8 TB/s per GPU1 |
| InfiniBand NDR (Quantum-2) | Server ↔ server, datacenter | Scale-out (clustering) | 400 Gb/s per port5 |
Why this decides how fast models train
As GPU counts climb into the thousands, the fraction of each step spent on communication rather than computation grows. A cluster with slow interconnects hits a wall: adding more GPUs stops making training meaningfully faster, because the extra chips spend their time waiting to synchronise. Fast interconnects push that wall much further out — which is precisely why NVLink bandwidth and InfiniBand data rates get headline billing every GPU generation, right alongside the raw compute numbers.
Takeaways
- NVLink = scale-up. Direct GPU-to-GPU links inside a server or rack; NVLink 5 reaches 1.8 TB/s per GPU, and NVSwitch turns dozens of GPUs into one.12
- InfiniBand = scale-out. A low-latency, RDMA-based datacenter fabric; Quantum-2 runs at 400 Gb/s NDR per port with in-network computing.5
- RDMA and GPUDirect let data move GPU-to-GPU across the network without touching a CPU — the trick that makes cluster training efficient.7
- They stack. NVLink inside the node, InfiniBand between nodes, NCCL routing the traffic — together they let thousands of GPUs behave like one machine.9
- The interconnect is a bottleneck, not a detail. Beyond a point, communication speed — not GPU count — sets how fast a large model trains.