NVIDIA and ETH Zurich Push Multi-GPU Communication Latency Toward the Speed of Light
Every µs Matters: Achieving Near Speed-of-Light Latency in GPU Collectives
As large language models (LLMs) continue to scale, GPU performance is no longer determined solely by compute throughput or memory capacity. While advances in model quantization, KV cache optimization, and increasingly powerful accelerators have significantly improved inference efficiency, another bottleneck has become impossible to ignore: inter-GPU communication latency.
The recently published paper, Every μs Matters: Achieving Near Speed-of-Light Latency in GPU Collectives, co-authored by NVIDIA and ETH Zurich, examines one of the most fundamental operations in distributed AI systems—GPU collective communication. Rather than focusing on increasing bandwidth, the researchers demonstrate that reducing synchronization overhead by only a few microseconds can dramatically improve large-scale AI inference and high-performance computing (HPC) workloads.
Their work introduces several synchronization techniques that push AllReduce latency remarkably close to the theoretical physical limit imposed by the underlying hardware.
🚀 Communication Has Become the New Bottleneck #
Distributed inference relies heavily on collective communication.
In Tensor Parallel (TP) deployments, every generated token requires GPUs to repeatedly exchange intermediate activations through AllReduce operations. During long-context inference, most GPU memory is occupied by the KV cache, reducing batch sizes and leaving each communication round to transfer only relatively small payloads.
Typical message sizes often range from only tens to hundreds of kilobytes.
For messages this small, bandwidth is rarely the limiting factor.
Instead, latency dominates.
The critical question shifts from:
How much data can be transferred per second?
to:
How long does every communication operation spend waiting before useful work can begin?
The paper demonstrates the significance of this distinction.
On a four-GPU GB200 system:
- Conventional NCCL Ring AllReduce requires approximately 11 μs
- The proposed communication kernel completes the same operation in roughly 2.37 μs
This represents nearly a 4.6× reduction in latency, bringing communication within approximately 7% of the theoretical speed-of-light limit imposed by the hardware interconnect.
Although the improvement appears small in absolute terms, inference workloads execute millions—or even trillions—of communication operations over time, allowing microsecond-level savings to accumulate into substantial performance and cost reductions.
⚙️ Why Memory Barriers Become Expensive #
After analyzing communication libraries including:
- NCCL
- NVSHMEM
- MSCCL++
- vLLM
the researchers identified one major source of latency:
Global memory barriers.
Traditional GPU collective communication relies on synchronization barriers to guarantee correctness.
A simplified workflow looks like this:
- GPU A writes data.
- GPU A notifies every other GPU.
- Every GPU waits until all notifications are received.
- The next communication stage begins.
This guarantees correctness but introduces unnecessary waiting.
On NVIDIA GB200 systems:
- A single global memory barrier can exceed 1 μs
- Some AllReduce implementations require two barriers
- Barrier synchronization alone may account for nearly 40% of total communication latency for small messages
The problem becomes increasingly severe as GPU counts grow.
Since every participant must synchronize with every other participant, coordination overhead grows much faster than the communication payload itself.
At scales of dozens of GPUs, synchronization delays become one of the largest contributors to overall inference latency.
🔄 Four Techniques That Eliminate Global Barriers #
The paper introduces four complementary synchronization mechanisms designed to remove explicit global barriers while preserving correctness.
Each addresses a different communication scenario.
LL Atomic Synchronization #
The LL (Low Latency) protocol targets extremely small messages.
Each communication slot occupies 16 bytes:
- 8 bytes for payload data
- 8 bytes for a validity flag
Both are transmitted together through a single atomic write.
Because the receiver can immediately determine whether incoming data is valid, no additional synchronization notification is required.
Advantages include:
- Extremely low latency
- Immediate validation
- No explicit synchronization messages
The tradeoff is reduced payload efficiency because part of every packet is reserved for status information.
Sentinel Synchronization #
Sentinel synchronization replaces explicit validity flags with predefined marker values.
The receive buffer is initialized using a special sentinel value—for example, a unique NaN bit pattern.
The receiver continuously polls the buffer:
- Sentinel unchanged → data has not arrived.
- Sentinel replaced → payload is ready.
Compared with LL mode, sentinel synchronization provides higher bandwidth efficiency because payloads contain only application data.
However, it introduces two practical constraints:
- Buffers must be reset before reuse.
- Valid application data must never equal the sentinel value.
This approach is particularly well suited for small-to-medium communication payloads.
Double Buffering with Bidirectional Communication #
Large messages are typically divided into multiple communication rounds.
Without additional protection, a faster GPU could overwrite shared buffers before slower GPUs finish processing previous data.
The paper eliminates this problem using:
- Bidirectional communication
- Double buffering
Two alternating buffers are maintained.
While one buffer is being transmitted, the other is being received.
Receiving the current communication round implicitly confirms that the peer has reached the same execution stage, removing the need for explicit global synchronization.
Instead of stopping every GPU at a barrier, synchronization naturally occurs through the communication process itself.
LL128 Hardware Atomic Algorithm #
Perhaps the paper’s most innovative contribution is the Two-Shot LL128 Atomic AllReduce algorithm.
Rather than synchronizing GPUs through software barriers, the algorithm leverages NVLink hardware atomic operations.
The process consists of two phases:
- ReduceScatter
- AllGather
Each 128-byte cache line contains:
- Data payload
- Hardware-managed atomic counter
Every GPU performs an atomic addition on both:
- Local data
- Shared completion counter
Once the counter equals the total number of participating GPUs, every contribution has been received.
Because atomic operations are implemented directly in hardware through NVLink and NVSwitch, synchronization occurs naturally without explicit barriers.
Compared with previous approaches, LL128 scales much more efficiently as GPU counts increase.
Design Limitations #
The LL128 algorithm is highly optimized but not universally applicable.
Current limitations include:
- Requires NVLink hardware atomics
- Depends on symmetric memory support
- Supports only selected reduction operations
- Primarily optimized for addition
- Floating-point atomics are not bitwise deterministic
For AI inference, where tiny numerical differences are generally acceptable, these tradeoffs are often worthwhile.
Scientific computing applications requiring strict reproducibility may instead prefer deterministic communication methods.
🧩 A Unified Communication API #
Rather than presenting isolated synchronization techniques, the paper combines them into a reusable communication framework.
The central abstraction is ncclLLBuffer, which hides protocol-specific implementation details while exposing simple communication primitives.
Key operations include:
| Primitive | Purpose |
|---|---|
| send | Write data to a peer buffer |
| recv | Poll until incoming data becomes valid |
| recvReduce | Receive and reduce multiple inputs directly in place |
| bcast | Broadcast data to all peers using multicast when available |
| reset / resetRange | Reinitialize communication buffers |
These building blocks allow developers to construct custom collective algorithms without manually implementing synchronization protocols or memory ordering semantics.
The paper demonstrates that an entire single-shot AllReduce implementation can be expressed in only a few dozen lines of CUDA code.
📊 AllReduce Algorithms #
Using this communication framework, the authors implement three optimized AllReduce variants.
| Algorithm | Best Use Case | Synchronization Strategy |
|---|---|---|
| Single-Shot LLBuffer | Ultra-small messages (<64 KB) | LL or Sentinel with Double Buffering |
| Two-Shot LLBuffer | Medium messages (64 KB–1 MB) | Sentinel with Double Buffering |
| Two-Shot LL128 Atomic | Small-to-medium messages across larger GPU clusters | Hardware Atomics with L2 Synchronization |
Each algorithm targets a different communication regime while minimizing synchronization overhead.
📈 Benchmark Results #
Near Speed-of-Light Latency #
Microbenchmarks conducted on GB200 NVL72 systems demonstrate substantial improvements.
On four GPUs:
- NCCL Ring: approximately 11 μs
- New communication kernel: 2.37 μs
This places measured latency only about 7% above the hardware’s theoretical physical limit.
At 64 GPUs, multicast-enabled variants remain within roughly 70% of the theoretical minimum, significantly outperforming conventional implementations.
Large Language Model Inference #
The researchers integrated their communication kernels into vLLM, evaluating several production-scale language models, including:
- Llama 3.1 70B
- DeepSeek-V3
- Qwen3-Next
Reported improvements include:
- 7–13% lower token latency on four-GPU Tensor Parallel deployments
- 9–11% lower latency across two-node, eight-GPU systems
- Reduced inference cost per million generated tokens
Although exact savings vary depending on workload characteristics, even small latency reductions become financially meaningful at trillion-token inference scales.
High-Performance Computing #
The communication improvements also benefit scientific computing.
Using NVIDIA’s distributed linear algebra library cuSOLVERMp, the researchers accelerated generalized symmetric eigenvalue solvers on the Alps supercomputer.
Consistent improvements were observed across multiple matrix sizes, illustrating that optimized collective communication extends beyond AI workloads into traditional HPC applications.
🔮 Implications for AI Infrastructure #
The paper focuses specifically on communication within a single NVLink domain and therefore does not address broader networking challenges such as cross-rack or cross-data-center communication.
Several techniques also depend on hardware features unique to modern NVIDIA systems, including:
- GB200 GPUs
- NVLink
- NVSwitch
- Hardware multicast
- Symmetric memory
- Hardware atomic operations
Nevertheless, the research highlights an important trend in AI infrastructure.
As advances in quantization, speculative decoding, and memory optimization continue reducing computational overhead, communication increasingly determines end-to-end inference performance.
Every transformer layer requires GPUs to exchange intermediate results.
Even a single additional microsecond, multiplied across billions of communication operations, directly affects:
- Token generation latency
- GPU utilization
- Cluster throughput
- Infrastructure efficiency
- Cloud inference costs
📖 Conclusion #
Every μs Matters demonstrates that future AI performance gains will not come exclusively from faster GPUs or larger memory systems. Instead, significant improvements can be achieved by fundamentally rethinking how GPUs communicate.
By eliminating expensive global memory barriers and introducing synchronization mechanisms based on hardware atomics, sentinel values, and lightweight communication primitives, NVIDIA and ETH Zurich show that collective communication latency can approach the physical limits of modern interconnects.
As distributed AI systems continue to scale, communication efficiency is becoming as critical as computational performance. The next generation of LLM infrastructure may depend not only on more powerful accelerators, but also on ensuring that dozens—or even hundreds—of GPUs spend less time waiting for one another and more time performing useful computation.