The chapter begins with a fundamental observation: when scaling beyond a single node, network communication transforms from an auxiliary mechanism into the critical path of performance. While on a single GPU the bottleneck is usually related to computation or memory, on a cluster of thousands of accelerators the dominant factor becomes how quickly data can be transferred between them.

The key idea of the chapter is to treat communication not as an «infrastructure detail» but as a full-fledged object of optimization on par with CUDA kernels. Network optimization can deliver a greater effect than compute optimization — especially at scale. This is why the chapter examines NCCL, NIXL, SHARP, RDMA and overlap strategies in detail.

In the chapter, the overlap principle is described as the foundation of all distributed training optimization. If a GPU must wait for all-reduce of gradients to complete before starting the next step, each step takes the sum of compute and communication times. If communication is launched asynchronously and overlapped with computation, the step time approaches the maximum of these two values rather than their sum.

PyTorch DistributedDataParallel implements this through wait-free backpropagation: gradients are split into buckets, and as each bucket becomes ready, an asynchronous all-reduce is launched on a separate CUDA stream. The main stream continues computing gradients for subsequent layers. The default bucket size is 25 MiB — a trade-off between network utilization and communication start latency.

The practical effect noted in the chapter: with proper tuning, most communication is hidden behind computation. The visible performance gain for a typical model is 20–30% of iteration time, and for larger models — significantly more.

The chapter emphasizes that overlap is impossible without asynchronous execution. CUDA streams provide the mechanism allowing different operations to execute in parallel. The key requirement is that communication operations (NCCL) use nonblocking calls that return control immediately. This allows the main stream to continue computing while NCCL works in the background.

In PyTorch DDP this mechanism is hidden from the user, but it is important to understand the principle: each gradient bucket is sent to a separate communication stream, while the main stream continues backward. If the user accidentally calls a synchronization (e.g., to get a tensor value), all overlap is destroyed.

Practical consequence: to preserve overlap, any implicit synchronizations in the hot loop should be avoided. The only synchronization should be at the very end of the iteration, when it is necessary to wait for all asynchronous work to complete before moving to the next step.

In the chapter, reducing the frequency and volume of communication is considered the second most important principle after overlap. Gradient accumulation reduces the number of all-reduce operations linearly with the number of accumulation steps. Compression and sparsification reduce the volume of data transferred per operation.

The trade-off of buckets is separately stipulated: very large buckets maximize bandwidth utilization but delay the start of communication. Very small buckets start earlier but generate more per-call overhead. The default size in PyTorch — 25 MiB — is an empirical balance, but for a specific model it may require tuning.

Practical consequence: if a model has very large layers, increasing the bucket reduces overhead. If a model has many small layers, decreasing the bucket allows communication to start earlier.

In the chapter, RDMA is described as a fundamental technology for multi-node communication. The key difference from TCP/IP: RDMA allows the network adapter to directly access the memory of a remote node without CPU involvement and without copying through system RAM. GPUDirect RDMA extends this to GPU memory — the NIC can write directly to the HBM of a remote GPU.

The practical effect — reducing latency from 20–50 microseconds (TCP/IP) to 1–5 microseconds (RDMA) and increasing throughput from ~100 Gb/s to 400–800 Gb/s per port. At the scale of thousands of GPUs, this delivers a multiplicative gain in gradient synchronization.

The chapter separately notes that in container environments (Docker, Kubernetes), NCCL may fail to access InfiniBand devices and quietly switch to TCP sockets. There are no errors, training proceeds — but throughput drops by an order of magnitude. This is one of the most insidious problems because it does not cause failures.

The chapter enumerates six typical pitfalls of multi-node training: wrong backend (Gloo instead of NCCL), NCCL version mismatch, TCP port exhaustion, insufficient network bandwidth, straggler nodes, and GPU memory fragmentation under UCX/RDMA.

Each pitfall has a characteristic symptom. Gloo instead of NCCL causes a 10–50× performance drop without any errors. NCCL version mismatch leads to hangs or silent fallback to a slow implementation. TCP port exhaustion manifests as handshake hangs without an explicit error. A straggler node slows down the entire cluster, since synchronous training is determined by the slowest participant.

Practical consequence: for diagnosing each pitfall, the chapter suggests a specific tool. For the backend — torch.distributed.get_backend(). For NCCL versions — torch.cuda.nccl.version(). For straggler nodes — dist.monitored_barrier with a timeout. For the network — monitoring via nvidia-smi dmon or ibstat.

In the chapter, NCCL is described as the foundation of distributed training in the NVIDIA ecosystem. The key feature of NCCL is automatic selection of the optimal data transmission path based on physical topology. The library itself determines whether to use NVLink, PCIe, InfiniBand or TCP, and builds the corresponding communication scheme.

Complementing NCCL, the chapter introduces NIXL (NVIDIA Inference Xfer Library) — a library for point-to-point transfers optimized for inference. If NCCL solves the «many-to-many» problem (all-reduce of gradients in training), then NIXL solves the «one-to-one» problem (KV cache transfer between prefill and decode in disaggregated inference).

Practical consequence: for training, NCCL is used; for inference with KV cache transfer — NIXL. Both libraries complement each other rather than replace one another.

The chapter examines four main NCCL algorithms. Ring provides perfect load balancing (each link transfers an equal volume), but latency grows linearly with the number of GPUs. Tree reduces in O(log N) steps, which is better for small messages, but not all links are evenly loaded. CollNet builds a two-level tree: local within a node (NVLink) and global between nodes (RDMA). PAT is a hybrid of ring and tree, splitting a message into fragments with overlap.

Practical consequence: NCCL selects the algorithm automatically based on message size and topology. For debugging, it can be overridden via NCCL_ALGO, but in production the automatic choice is usually optimal.

In the chapter, topology awareness is described as one of the main reasons NCCL is effective. The library analyzes the physical connection scheme of GPUs and builds a hierarchical communication pattern. If GPU 0–1 are connected via NVLink and GPU 2–3 via another NVLink, but only PCIe connects the pairs, NCCL first reduces within pairs (fast NVLink), then exchanges between pairs via PCIe (slow, but only part of the data).

Practical consequence: for diagnosing scaling problems, nvidia-smi topo -n is used. If scaling is worse than expected, the first hypothesis is that traffic is going over slow paths, for example, through PCIe instead of NVLink or through Ethernet instead of InfiniBand.

The chapter examines four key pitfalls related to NCCL: creating communicators too often, creating and destroying communicators on every iteration, disabling features via environment variables without re-enabling them, and ignoring NCCL warnings.

Creating an NCCL communicator is an expensive operation. If code initializes and destroys a communicator on every iteration, the overhead can exceed the training time itself. The correct approach is to initialize once outside the loop.

The chapter separately notes that NCCL environment variables should be left at their default values unless there is a specific reason to change them. Some variables, such as NCCL_P2P_DISABLE or NCCL_SHM_DISABLE, can be used for debugging, but left enabled in production they noticeably reduce performance.

In the chapter, SHARP is described as a technology that moves part of the collective operation into the network. Instead of each GPU receiving data from all others and performing the reduction itself, the switch aggregates partial results. This reduces both SM load and traffic volume.

The practical effect noted in the chapter: NVIDIA reports 2–5× acceleration of all-reduce at large scales. The gain depends on cluster size: at 2–4 GPUs it is negligible, at 72+ GPUs within a rack it is substantial. SHARP is one of the key innovations NVIDIA gained through its Mellanox acquisition.

The chapter also notes NCCL persistent user buffers (user buffer registration), allowing collectives to work directly with tensors without intermediate copying. This is especially important for achieving the best paths through SHARP on both NVLink (NVLS) and InfiniBand.

In the chapter, NIXL is described as an evolutionary step for inference. Traditional NCCL is optimized for collective operations (all-reduce of gradients), but inference requires a different pattern: transferring KV cache between specific GPUs in disaggregated prefill-decode architecture.

NIXL provides point-to-point transfers with minimal latency and automatically selects the transport — NVLink for intra-node, InfiniBand/RDMA for inter-node, GPUDirect Storage for disk operations. The asynchronous API with callbacks allows overlapping transfer with computation.

Practical consequence: in disaggregated inference, prefill is performed on some GPUs, decode on others, and KV cache is transferred via NIXL. This allows independent scaling of prefill and decode, as well as optimizing each stage for its own characteristics.

The chapter provides a systematic comparison of NCCL and NIXL. NCCL is oriented toward «many-to-many» collectives with synchronous semantics (all participants must reach the call). NIXL is oriented toward point-to-point transfers with asynchronous semantics (one initiator, one or more receivers).

NCCL is used in training for all-reduce of gradients. NIXL is used in inference for KV cache transfer and model shards. NCCL is optimized for small and medium tensors, NIXL — for large data blocks (hundreds of megabytes). NCCL is integrated into PyTorch DDP, Horovod and other training frameworks. NIXL is integrated into NVIDIA Dynamo, vLLM via LMCache, and other inference engines.

Practical consequence: for training — NCCL, for inference — NIXL. Both libraries can be used in one application if it includes both training and inference.

The chapter mentions ConnectX-8 and BlueField-4 as promising developments. By 2026, ConnectX-8 (800 Gb/s) and BlueField-4 DPU have become standard for new deployments. ConnectX-8 provides double the throughput of the previous generation, which is critical for disaggregated inference with large KV caches.

The chapter mentions that the NVLink Switch System can combine up to 576 GPUs into a single domain. With the release of Vera Rubin, NVLink 6 doubles throughput to 3.6 TB/s per GPU, and the domain expands to an even larger number of GPUs. This allows TP and EP across even more GPUs without leaving the NVLink fabric.

The chapter mentions NVIDIA Dynamo as an example of NIXL usage. By 2026, Dynamo has become the standard for deploying disaggregated inference in Kubernetes. vLLM, SGLang, TensorRT-LLM — all support integration with NIXL via LMCache or their own connectors.

The chapter does not mention PDL. This is a technology that appeared in Hopper and was developed in Blackwell, allowing one kernel to signal the completion of its prologue and launch a dependent kernel before the first fully completes. PDL complements overlapping communication with computation, reducing time between successive kernel launches.

The chapter does not mention the integration of Groq LPU into the Vera Rubin platform. This is recognition that a single accelerator type is insufficient for reasoning workloads, where test-time scaling multiplies inference load. Heterogeneous computing is becoming a new trend, and this affects communication strategies.