ByteDance's ByteExpress Makes RDMA Scale for AI Inference

ByteDance’s ByteExpress Makes RDMA Scale for AI Inference

As AI models grow, GPU computing power is no longer the only factor determining application performance. In distributed inference and training systems, network communication can become a significant bottleneck, adding latency and consuming CPU resources that could otherwise support computation.

Remote Direct Memory Access (RDMA) offers a way to reduce this overhead by allowing network adapters to transfer data directly between registered memory regions without requiring the conventional kernel networking path for every operation. However, replacing TCP with RDMA alone does not solve the practical challenges of large-scale machine learning infrastructure.

A research paper titled Deploying RDMA for Large-Scale Inference and Training Workloads, published by researchers associated with ByteDance and academic institutions, describes the company’s experience deploying RDMA in production parameter-server systems.

The paper presents ByteExpress (BE), a software communication layer integrated with ByteDance’s existing Archon RPC framework. Its design combines RDMA-aware memory allocation, zero-copy data transfer, connection multiplexing, layered flow control, and application-level failure recovery.

In production inference, ByteExpress reduced average parameter-fetch latency by 35.7%, P99 latency by 43.7%, and total machine CPU utilization by 10.69% compared with kernel TCP. In offline training, it reduced P99 parameter-push latency by up to 74%, with no PFC pause events observed in the reported deployment.

The central lesson is that production RDMA is not simply a hardware transport choice. It requires coordinated design across memory management, network congestion control, RPC semantics, and service availability.

🚀 1. Why Replacing TCP with RDMA Is Not Enough

#

ByteDance’s recommendation systems use a Parameter Server (PS) architecture in which model parameters are distributed across multiple servers. Computing workers retrieve or update the parameter shards they need over the network.

Two important workloads use this communication pattern:

• Online inference: Ranker services fetch model parameters from parameter servers while processing requests.
• Offline training: Trainer workers communicate with parameter servers to push model updates and retrieve parameter data.

In both cases, networking sits directly on the critical path. Network delays can increase inference response times, while inefficient parameter transfers can reduce training throughput.

Traditional TCP networking introduces processing overhead from kernel networking, memory copies, packet handling, and interrupt or scheduling activity. As accelerator performance increases, these costs can consume a growing share of the system’s available CPU capacity.

RDMA can reduce many of these costs, but a straightforward replacement of TCP with RoCE introduces several additional engineering challenges.

Challenge 1: Variable-size messages

#

Machine-learning applications generate messages ranging from kilobytes to megabytes. Their exact sizes may not be known until request processing has advanced sufficiently.

RDMA requires the participating memory regions and transfer parameters to be established according to the applicable operation and access rules. Allocating and registering a separate memory region for every message would impose significant CPU overhead and increase pressure on NIC memory-registration resources.

Challenge 2: Large numbers of concurrent connections

#

Production inference services can maintain many simultaneous connections between rankers and parameter servers.

A dedicated Reliable Connection (RC) Queue Pair (QP) for every logical connection can create substantial resource overhead. NICs have finite internal resources for managing active queue pairs, and excessive QP counts can degrade performance.

Concurrency also creates traffic bursts. When many workers request data from the same parameter server at once, the destination can become congested even when the cluster’s total network bandwidth appears sufficient.

Challenge 3: Incast congestion

#

Incast occurs when many senders transmit data to one receiver or a small set of receivers simultaneously.

This pattern is common in distributed machine learning, where synchronized workers can generate substantial traffic toward a parameter server over a short period.

Hardware congestion control may react after congestion signals appear, but that response does not necessarily prevent an initial traffic burst from overwhelming switch buffers. The resulting congestion can trigger Priority-based Flow Control (PFC) pauses and increase tail latency.

Challenge 4: Application-level reliability

#

Reliable Connection RDMA provides transport-level reliability under its defined operating conditions. It does not, on its own, decide what the application should do when a remote server crashes, a network interface fails, or a service becomes unavailable.

Production services still need request deadlines, retries, replica selection, connection recovery, and fallback behavior.

ByteExpress addresses these problems by coordinating the RDMA data path with Archon RPC rather than treating RDMA as a standalone replacement for the complete communication stack.

For background on memory registration and RDMA operations, see KAD8’s explanation of how Remote Direct Memory Access works.

🏗️ 2. ByteExpress Architecture: RDMA Beneath Archon RPC

#

ByteExpress is designed as a communication layer beneath Archon RPC. It handles low-level transport operations while leaving application-oriented request semantics and much of the business-level recovery policy to the RPC framework.

The architecture separates responsibilities into three layers.

+------------------------------------------------------+
|                  Archon RPC                          |
| Timeouts | Retries | Replicas | TCP Fallback         |
| Request Semantics | Service-Level Recovery           |
+------------------------------------------------------+
                         |
                         v
+------------------------------------------------------+
|                 ByteExpress (BE)                     |
| Socket API | bemalloc | Engines | Schedulers         |
| Receive Credits | Software Send Windows              |
+------------------------------------------------------+
                         |
                         v
+------------------------------------------------------+
|                 RNIC Hardware                        |
| RDMA Verbs | Reliable Connections | DCQCN            |
| Direct Memory Transfers | Completion Queues          |
+------------------------------------------------------+

2.1 Socket: A logical communication endpoint

#

ByteExpress exposes a socket-like interface to the upper layer. A Socket represents a logical communication channel associated with an RPC connection.

This abstraction allows application code to use a familiar communication model without managing every low-level RDMA resource directly.

A logical Socket does not necessarily own an independent hardware Queue Pair. Multiple Sockets can share an Engine, reducing the number of hardware connections the system must maintain.

2.2 Engine: The hardware connection management unit

#

An Engine is bound to a single RC Queue Pair in the described architecture. It manages the transport state and schedules operations submitted by its associated logical Sockets.

Multiple Sockets targeting the same remote endpoint can be multiplexed over one Engine. This reduces QP proliferation and allows the software layer to coordinate transmission across different logical connections.

However, multiplexing also introduces a scheduling problem: a large transfer from one Socket should not monopolize the shared Engine and delay smaller, latency-sensitive requests.

ByteExpress therefore combines QP sharing with message fragmentation and a scheduling policy designed to reduce head-of-line blocking.

2.3 bemalloc: Memory designed for RDMA

#

The bemalloc allocator provides memory that is suitable for RDMA operations and manages registration at a larger allocation granularity.

Ordinary allocation APIs do not automatically provide the registered-memory state required by RDMA. ByteExpress integrates allocation and memory registration so that the transport layer can reuse registered buffers rather than paying the full registration cost for every message.

This shared buffer-management model is the foundation for the zero-copy transfer paths described in the paper.

2.4 Three independent control mechanisms

#

ByteExpress coordinates three mechanisms that address different constraints:

These mechanisms are complementary rather than interchangeable.

Receive credits protect receive-side capacity. Software send windows govern how much data the software allows to remain in flight. DCQCN adjusts sending rates based on congestion information.

PFC provides an additional link-layer mechanism in networks configured to use it. The objective is not merely to make RDMA transfers fast, but to control how traffic enters the network under realistic production workloads.

🧠 3. bemalloc: Reducing Memory-Registration Overhead

#

RDMA memory registration allows a network adapter to access specified memory regions under controlled permissions. Registering memory requires setup work involving the operating system, the RDMA stack, and the NIC.

If an application allocates many short-lived buffers and registers each one separately, registration and deregistration overhead can become a significant cost.

ByteExpress addresses this problem through bemalloc, an RDMA-aware allocator built on rpmalloc.

3.1 Register large memory blocks instead of individual messages

#

The allocator obtains relatively large blocks of memory from the operating system. The paper’s example uses blocks of approximately 4 MB.

Instead of registering every application buffer separately, the allocator registers the larger memory block as an RDMA memory region. It then divides that registered space into smaller allocations for application use.

The resulting workflow is:

1. Request a large block of memory from the operating system.
2. Register the block with the RDMA subsystem.
3. Divide it into smaller buffers.
4. Allocate and release these buffers through the allocator.
5. Reuse registered memory as new messages are processed.

When the existing pool has enough capacity, ordinary allocation and deallocation can reuse registered blocks without invoking memory-registration APIs for every message.

Additional registrations are required when the pool grows, but their cost is amortized across many individual allocations.

3.2 Why allocator integration matters

#

This approach makes buffer allocation part of the transport design rather than an independent application concern.

The application can allocate a buffer from the RDMA-aware pool and pass it into the communication layer without requiring a separate registration step for each message.

The source paper reports that this design helped keep memory usage around 2.2 GB on the evaluated inference nodes under the described production workload.

That figure should not be interpreted as a universal memory requirement for RDMA inference. Actual memory usage depends on traffic patterns, buffer sizes, concurrency, workload characteristics, and pool configuration.

The more general principle is that RDMA efficiency depends on the entire buffer lifecycle, including allocation, registration, transfer, ownership, and reclamation.

🔄 4. Zero-Copy Transfers and Buffer Ownership

#

Memory registration alone does not eliminate all data movement. A communication stack can still copy payloads between application buffers and intermediate transport buffers.

ByteExpress instead coordinates buffer ownership so that received data can be placed directly into memory allocated by bemalloc.

4.1 Direct placement of received data

#

In a conventional networking path, data may pass through several buffers before reaching the application. Each intermediate copy consumes CPU cycles and memory bandwidth.

In the described ByteExpress design, the receiver provides a suitable registered buffer for the incoming message. The transport layer places the data into that buffer and then transfers ownership to the worker thread.

The worker can process the received data directly, retain it for later use, or pass it to another thread without an intermediate transport-to-application payload copy.

This design reduces data movement and simplifies the relationship between the application and transport layers.

4.2 Ownership is part of the API contract

#

Zero-copy designs require clear ownership rules.

The system must establish which component is allowed to access or release each buffer at a particular point in its lifecycle. A buffer cannot be returned to the allocator while the application is still using the contents, and the transport cannot safely overwrite a buffer that remains owned by an active consumer.

Consequently, zero-copy is not merely an optimization applied to a single memory copy. It changes how application code, transport operations, and memory management coordinate access to the data.

ByteExpress integrates those responsibilities through its allocator and communication abstractions.

📦 5. Large-Message Transfers Using RDMA WRITE

#

Small messages and large payloads create different transport requirements. Large messages may consume substantial bandwidth and monopolize a shared connection if they are handled as a single uninterrupted transfer.

ByteExpress uses a fragmented RDMA WRITE pipeline for large messages.

5.1 Step-by-step transfer process

#

The transfer begins with a control-plane exchange that establishes the destination buffer.

Step 1: Request a remote buffer

The sender issues a WriteRequest to the receiving endpoint.

Step 2: Allocate registered memory

The receiver allocates a suitable buffer from bemalloc and returns a WriteResponse. The response includes the remote address and the remote key (rkey) needed to authorize the applicable RDMA operation.

Step 3: Split the payload into fragments

The sender divides the large message into smaller pieces. These fragments can then be scheduled alongside operations belonging to other logical Sockets.

Step 4: Transfer the fragments

The sender issues RDMA WRITE operations to place each fragment directly into the registered destination buffer.

Step 5: Signal completion

The final fragment uses WRITE_WITH_IMM, or RDMA WRITE with Immediate, to provide an immediate notification value to the receiver. This allows the receiver to observe the operation through the appropriate completion mechanism and process the completed message.

The remote-memory address and rkey are security-relevant transport parameters. They must be exchanged and used according to the RDMA protocol and memory-access configuration.

5.2 Why fragmentation improves scheduling

#

Fragmentation enables the software layer to interleave transmission from different Sockets rather than allowing one large message to occupy the shared Engine for an extended sequence of operations.

This matters when a server handles a mixture of large parameter transfers and small latency-sensitive requests.

Fragmentation does introduce overhead: smaller fragments can increase scheduling work and generate more transport operations. The fragment size must therefore balance scheduling responsiveness against efficient link utilization.

⚖️ 6. QP Multiplexing and Head-of-Line Blocking

#

RDMA Queue Pairs consume hardware and software resources. Creating a separate RC QP for every application-level connection can lead to resource pressure, especially in systems with thousands of concurrent logical connections.

ByteExpress uses QP multiplexing to reduce that pressure.

6.1 Multiplex logical Sockets over a shared Engine

#

Multiple Sockets targeting the same remote endpoint can share a single Engine and its associated RC QP.

This approach reduces the number of Queue Pairs required and allows the software scheduler to coordinate operations from multiple logical connections.

However, QP sharing alone can create head-of-line blocking. If a large message is submitted as a long sequence of fragments and the scheduler cannot interleave other traffic, smaller messages may have to wait behind it.

6.2 Fragment scheduling reduces blocking

#

ByteExpress breaks large transfers into fragments and uses round-robin scheduling to interleave operations from different Sockets between fragments.

The paper describes fragment sizes tuned to the link speed, including approximately 2 KB for 25 Gb/s links and at least 8 KB for 100 Gb/s links.

These values reflect the reported configuration, not universal optimums. Appropriate fragment sizes depend on network bandwidth, transport overhead, workload composition, and the desired balance between throughput and latency.

The design principle is more general: connection multiplexing needs a scheduling policy that protects small messages from being trapped behind large transfers.

🌐 7. Controlling Incast with Software Send Windows and DCQCN

#

One of the more important findings from ByteExpress concerns training workloads, where many workers can push model updates toward parameter servers at nearly the same time.

This synchronized traffic can generate incast congestion even when the RDMA transport and individual network links are operating correctly.

7.1 Why hardware congestion control may not be sufficient

#

DCQCN is a congestion-control mechanism for RoCE that adjusts sending rates in response to congestion signals, typically using Explicit Congestion Notification (ECN).

Its purpose is to reduce the rate at which senders inject traffic when the network indicates congestion.

However, rate adjustment takes time to affect the traffic already in flight or being injected by many senders. A sudden incast can therefore create a burst that overwhelms switch buffers before the congestion-control response takes effect.

In networks using PFC, buffer pressure may trigger pause frames. These pauses can propagate congestion effects and increase tail latency.

7.2 Add a software send window

#

ByteExpress introduces a software send window for each Engine. The window imposes a limit on the amount of data that may remain in flight.

Rather than relying on congestion control alone to react after a burst is underway, the software layer limits the amount of outstanding data injected by that Engine.

This gives the transport an additional means of controlling burst volume before the network becomes severely congested.

7.3 Three mechanisms with different roles

#

The complete approach combines three levels of control:

The mechanisms address separate parts of the problem. Receive credits do not replace congestion control, while DCQCN does not directly enforce a fixed per-Engine outstanding-data limit.

According to the paper, enabling software send windows reduced training parameter-push P99 latency by up to 74% compared with kernel TCP, while the reported deployment observed no PFC pause events.

These results are important because they connect a transport-level control mechanism to a production workload outcome. However, they should be interpreted in the context of the evaluated system, workload, and configuration rather than as a guarantee that software send windows will eliminate PFC events in every RDMA cluster.

For a related example of networking designs developed specifically for AI infrastructure, see KAD8’s analysis of ByteDance’s veRoCE approach.

🛡️ 8. High Availability Requires Recovery Above RDMA

#

An RDMA RC connection provides reliable transport behavior while the underlying connection and participating endpoints operate within the protocol’s defined conditions. It cannot independently recover a distributed service from every host failure or application-level fault.

ByteExpress therefore integrates with Archon RPC to handle service availability.

8.1 Detect connection failures

#

The Engine periodically transmits keepalive sequences. If the expected progress stops, the system can mark the connection as faulty and initiate recovery.

The distinction between connection health and request health matters. A connection can fail even when an application request has not yet exceeded its own deadline, and an individual request can time out even when the transport remains operational.

8.2 Use separate connection and request timeouts

#

The described implementation uses two timeout levels:

These are deployment-specific settings, not general RDMA defaults. They reflect different failure-detection and application-response requirements.

Separating the timeouts allows the RPC layer to reason about request deadlines without assuming that transport-level connection monitoring is sufficient.

8.3 Apply a layered failover strategy

#

The reported recovery hierarchy includes:

1. Port-affinity switching: Attempt to move communication to an alternate port or path when the current one fails.
2. Replica migration or reselection: Redirect service requests to a healthy parameter-server replica where the application architecture supports it.
3. TCP fallback: Return to the conventional TCP transport when the RDMA path cannot provide service.
4. Failure isolation: Blacklist nodes that exceed configured failure thresholds so that new traffic can avoid repeatedly targeting unhealthy instances.

The exact sequence depends on the failure type and the capabilities of the deployment. Not every fault can be recovered through port switching, and replica failover requires suitable replicas and application-level routing logic.

The broader architectural point is that transport reliability and service availability are different concerns. The transport can report failure, but the RPC layer has the semantic information needed to decide whether to retry, select a replica, or fall back to another transport.

📊 9. Production Results: Inference Latency and Training Tail Latency

#

The ByteExpress paper reports production evaluations for online inference and offline training. The results compare ByteExpress with kernel TCP in ByteDance’s parameter-server architecture.

9.1 Online inference results

#

The reported deployment included 300 parameter servers and 124 rankers.

The reductions in both average and P99 latency indicate that the deployment improved typical parameter-fetch performance as well as the slowest portion of the measured request distribution.

The CPU reduction is also significant from a system-efficiency perspective. Less processing spent on communication can leave more host resources available for application logic and other services.

These figures apply to the reported deployment and measurement conditions. They do not establish the same percentage improvements for all inference architectures or all TCP and RDMA configurations.

9.2 Offline training results

#

The reported training deployment included 124 parameter servers and 346 trainers.

The training result highlights the importance of managing bursts rather than relying only on peak bandwidth.

A network may have high nominal throughput yet still produce poor application performance when synchronized workers overload a destination or trigger flow-control pauses. By regulating outstanding data at the software layer, ByteExpress seeks to make the traffic pattern more manageable before hardware congestion signals require a response.

The absence of observed PFC pauses is specific to the evaluated deployment. It should not be interpreted as proof that PFC can be removed from any network using ByteExpress.

9.3 Overall results at a glance

#

Source: Deploying RDMA for Large-Scale Inference and Training Workloads.

🧵 10. Multithreaded Scaling and Resource Isolation

#

At high connection counts, transport performance depends not only on NIC capabilities but also on how software resources are allocated among CPU threads.

Shared queues and heavily contended locks can introduce synchronization overhead, undermining the benefits of RDMA’s direct data-transfer mechanism.

The paper highlights Thread Domains (TDs) and per-thread resources, including Completion Queues (CQs) and Shared Receive Queues (SRQs), as important components of multithreaded RDMA scaling.

The underlying objective is to reduce unnecessary contention in the driver and transport path while distributing work across threads.

A scalable implementation must balance resource reuse with isolation. Sharing every queue may create lock contention, while allocating a fully independent set of resources for every logical connection may cause QP and memory overhead to grow too quickly.

ByteExpress addresses this broader design tension by combining QP multiplexing, resource management, and scheduling. The optimum balance depends on the concurrency and traffic patterns of the application.

✅ 11. Key Lessons for Distributed AI Systems

#

ByteExpress demonstrates that high-performance RDMA deployments depend on several cooperating design decisions.

1. Treat buffers as part of the transport contract. An RDMA-aware allocator such as bemalloc amortizes memory-registration costs and lets application and transport layers coordinate buffer ownership. Zero-copy performance depends on the complete lifecycle, not just avoiding one memcpy().

2. Separate receive readiness from congestion control. Receive credits, software send windows, DCQCN, and PFC serve different purposes. Combining them allows the system to manage receiver capacity, bound outstanding traffic, react to congestion, and retain link-level protection where configured.

3. Reduce connection overhead without creating head-of-line blocking. Multiplexing logical Sockets over shared Queue Pairs reduces QP proliferation, while message fragmentation and scheduling allow competing requests to make progress.

4. Keep business-level recovery in the RPC layer. RDMA is a high-performance data path, not a complete service-availability framework. Request deadlines, retries, replica routing, unhealthy-node isolation, and TCP fallback need to be coordinated with application semantics.

5. Measure production behavior instead of relying only on link throughput. Average latency, P99 latency, CPU utilization, memory footprint, and congestion events reveal different aspects of performance. A successful optimization must improve the application workload without introducing unacceptable resource or reliability trade-offs.

🏁 Conclusion

#

ByteDance’s ByteExpress offers a practical example of how to deploy RDMA in a large-scale distributed machine-learning system without treating it as a simple replacement for TCP.

Its design combines a registered-memory allocator, zero-copy buffer ownership, fragmented RDMA WRITE operations, QP multiplexing, and separate mechanisms for receiver readiness and congestion control. Archon RPC supplies the application-level recovery functions needed to handle failures beyond the scope of a reliable transport connection.

The reported production results—35.7% lower average inference parameter-fetch latency, 43.7% lower P99 latency, 10.69% lower machine CPU utilization, and up to 74% lower training parameter-push P99 latency—show the potential gains of this coordinated approach.

The wider lesson for AI infrastructure is clear: RDMA performance comes from co-designing the transport, buffer lifecycle, congestion controls, and service-recovery logic around actual workload behavior. Hardware acceleration establishes the opportunity, but software architecture determines how effectively that opportunity translates into production performance.