DistDPU: Scaling AI Cloud Networking Without Full DPUs

DistDPU: Scaling AI Cloud Networking Without Full DPUs

Large-scale AI training and inference are pushing data center networking toward terabits per second (Tbps) of bandwidth per server. As GPU clusters expand, their networking infrastructure must support high-throughput collective communication, including AllReduce and All-to-All, while maintaining low latency, strong tenant isolation, and predictable performance.

However, a fundamental scaling problem is emerging: network bandwidth requirements are growing much faster than the computational resources required to manage the network.

Traditional data processing units (DPUs) combine control-plane processing, network virtualization, security enforcement, and high-speed packet forwarding on a single system-on-chip (SoC). Scaling bandwidth by installing additional full-featured DPUs consequently duplicates processors, memory, firmware, and management software—even when the additional control-plane resources remain largely idle.

To address this inefficiency, researchers from Fudan University, Tencent, and the University of Michigan developed DistDPU, a disaggregated DPU architecture for high-performance and cost-efficient AI clouds. The work was accepted at ACM SIGCOMM 2026, a leading computer networking conference.

The core idea is to separate the DPU into two components: an Orchestration Module (OM), which handles centralized control and management, and lightweight Execution Modules (EMs), which handle high-speed packet processing and data forwarding.

Instead of purchasing another complete DPU whenever a server needs more network bandwidth, operators can add execution modules while retaining a shared control module.

According to the research team’s reported production results, DistDPU has operated for more than 20 months, supporting over 10,000 GPUs and delivering up to 3.6 Tbps of aggregate bandwidth per server. Compared with stacking multiple NVIDIA BlueField-3 DPUs, the architecture reduces single-node hardware costs by up to 77.1% and power consumption by up to 59.3%, while maintaining performance close to commercial hardware baselines.

The design offers a different way to scale the networking infrastructure behind AI clouds: expand the data plane independently without proportionally expanding the control plane.

🧩 The Architectural Problem: Control and Data Planes Scale Differently

#

Traditional cloud networking and AI cloud networking exhibit significantly different traffic patterns. Understanding that distinction explains why a monolithic DPU can become an inefficient unit of bandwidth expansion.

Why traditional DPU stacking becomes inefficient

#

Conventional cloud workloads often generate numerous short-lived connections. Creating and destroying sessions, applying access-control policies, maintaining connection state, and processing exceptional conditions can impose substantial control-plane overhead.

In such environments, packaging management processors and data-forwarding engines into a single DPU makes practical sense. Each device provides both the computing resources needed to manage network functions and the hardware needed to process packets.

AI workloads have different characteristics.

Distributed training jobs may maintain long-lived, high-bandwidth connections between GPUs. Collective operations move large amounts of tensor data across the cluster, while inference workloads can generate sustained traffic between computing nodes and memory-intensive services.

As network interfaces scale from hundreds of gigabits per second toward multiple terabits per second, the forwarding hardware must process increasingly large volumes of traffic. Yet the number of virtual network configurations, management operations, and long-lived connections does not necessarily increase at the same rate.

This creates a growing mismatch between control-plane capacity and data-plane demand.

The essential problem is not that control-plane processing is unnecessary. It is that its resource requirements do not necessarily grow in proportion to the network bandwidth being added.

Three problems with stacking complete DPUs

#

Installing multiple full-featured DPUs can increase aggregate throughput, but it introduces three major costs.

Hardware expenditure: Each additional DPU includes a processor, memory, and management infrastructure, even if only a fraction of its control-plane capacity is needed.

Power consumption: The additional SoCs and supporting components consume power independently of whether their management resources are heavily utilized.

Operational complexity: Every additional DPU creates another managed device with firmware, configuration, monitoring, debugging, lifecycle management, and potential failure modes.

These problems become especially pronounced when several DPUs must be coordinated within the same server.

An alternative is to use less expensive network interface cards (NICs) for forwarding and return control logic to the host CPU. Although this can reduce hardware costs, it introduces different trade-offs.

In bare-metal cloud environments, tenants may have extensive privileges over their servers. Moving network management and isolation functions onto the tenant-controlled host can weaken the security boundary between tenant workloads and cloud infrastructure unless additional isolation mechanisms are provided.

AI clouds therefore need an architecture that combines high bandwidth, strong DPU-style isolation, unified virtual private cloud (VPC) management, and support for bare-metal, virtual machine, and container workloads.

DistDPU addresses this requirement by separating the management resources from the forwarding resources instead of removing the control plane altogether.

For broader context on the role of network interfaces in distributed AI systems, see KAD’s guide to AI cluster network architecture and GPU communication.

🏗️ DistDPU Architecture: One Brain, Multiple Forwarding Modules

#

DistDPU introduces a two-part architecture that separates orchestration from execution.

The Orchestration Module provides centralized control, while multiple Execution Modules scale the network data path. Together, these components present a unified logical DPU to the existing cloud management platform.

Orchestration Module: The control-plane brain

#

The OM contains a full-featured SoC processor and runs the software responsible for configuring and managing the execution modules.

Its responsibilities include:

• Translating cloud networking policies into device configurations and forwarding rules.
• Managing device virtualization and the lifecycle of execution modules.
• Selecting RDMA congestion-control algorithms and distributing configuration parameters.
• Coordinating VPC isolation, access-control policies, network address translation, and encapsulation.
• Collecting monitoring information and handling device faults.
• Integrating with the cloud provider’s existing management and orchestration systems.

The architecture requires only one primary OM per server, rather than one complete control-plane processor for every additional forwarding module.

The OM is therefore scaled according to control-plane requirements, not directly according to aggregate packet-forwarding bandwidth.

Execution Module: The high-speed forwarding muscle

#

An EM is a lightweight forwarding unit designed to handle packet processing without carrying the full management-oriented SoC of a traditional DPU.

Its responsibilities include:

• High-speed Ethernet and RDMA packet transmission and reception.
• Packet classification, forwarding-table lookup, encapsulation, and address rewriting.
• Hardware-assisted PCIe device virtualization.
• Local congestion feedback and multipath traffic distribution.
• Hardware execution of data-path operations that require predictable latency and throughput.

Additional EMs can be installed as a server’s bandwidth requirements increase.

This approach avoids repeatedly purchasing management processors and memory that would otherwise sit underutilized.

Integrating the modules into a server

#

A typical DistDPU server contains one full-featured network card with an OM and a local EM, together with one or more expansion cards containing additional EMs.

The modules communicate through an internal management fabric, allowing the OM to configure and monitor the full set of forwarding devices.

From the cloud management platform’s perspective, the arrangement appears as one logical DPU. The software does not need to manage every EM as an independent DPU with its own full management stack.

From the tenant’s perspective, standard PCIe device interfaces preserve compatibility with existing workloads. This abstraction is intended to support bare-metal servers, virtual machines, and containers without requiring changes to the applications running above the DPU.

The result is a hierarchical design: centralized orchestration controls multiple distributed forwarding engines, while the high-speed data path remains in hardware.

⚙️ Core Design Innovations: Virtualization, RDMA, and Forwarding

#

Separating control and execution introduces engineering challenges. The OM must coordinate multiple devices without becoming a bottleneck, while EMs must retain the performance and security properties expected from conventional DPUs.

DistDPU addresses these requirements through functional partitioning, hardware-assisted virtualization, optimized RDMA processing, and a dedicated management fabric.

Functional partitioning between OM and EM

#

The division of responsibility follows a central rule: operations that scale with packet traffic belong in the data path, while global policy decisions and management functions belong in the control plane.

This partitioning makes it possible to increase forwarding capacity without proportionally increasing the number of control-plane processors.

It also preserves an important operational boundary: the cloud platform continues to express policies in terms of logical networking services rather than managing a collection of unrelated forwarding devices.

SmartEP: Software-defined PCIe device virtualization

#

SmartEP is a central component of DistDPU’s device-virtualization architecture.

PCIe devices normally expose physical functions (PFs) and virtual functions (VFs) to host software. Providing a flexible virtual-device configuration can require substantial control logic, but executing data transfers through that same control path would introduce unnecessary overhead.

SmartEP separates PCIe control operations from data-path access.

A lightweight microcontroller in the EM manages device presentation and can dynamically emulate large numbers of virtual PCIe network interfaces. The OM determines the desired device configuration and distributes it to the appropriate EM.

Critical operations, such as doorbell access and packet-data transfers, use hardware-accelerated paths rather than repeatedly traversing the centralized control plane.

This separation allows device configuration to remain flexible while maintaining high-speed data access.

For cloud providers, it also helps preserve familiar virtualization semantics across bare-metal, virtual machine, and container deployment models without requiring a separate full DPU for every device configuration.

Direct Offset Translation for GPUDirect RDMA

#

RDMA is a core technology for AI clusters because it enables direct memory transfers between systems with minimal CPU involvement. GPUDirect RDMA extends this capability to GPU memory, reducing unnecessary data movement through host memory.

Virtualized environments introduce additional address-translation overhead. Address mappings may span many memory regions, and translation caches can become less effective when workloads access numerous GPU buffers.

DistDPU introduces Direct Offset Translation (DOT) to reduce this overhead.

Rather than relying exclusively on repeated fine-grained address translation, DOT uses range mappings that associate a base address with an offset. This allows address calculations to be performed more efficiently for accesses within a mapped region.

The goal is to reduce translation-cache pressure and make GPU-direct transfers more efficient in virtualized environments.

The benefit depends on workload characteristics, memory-region organization, and mapping behavior. DOT is designed to improve an important part of the RDMA path rather than eliminate every source of virtualization overhead.

Container-friendly RDMA address management

#

Container workloads introduce their own device-lifecycle challenges.

When Kubernetes restarts a container, the software environment may be recreated even though the underlying server and physical network devices remain active. If global identifiers associated with RDMA devices change unexpectedly, applications and orchestration systems may encounter unnecessary reconfiguration.

DistDPU maintains stable global identifier (GID) mappings at the EM layer, helping preserve consistent RDMA addressing across container restarts.

This approach reduces dependence on the lifecycle of individual containers and simplifies integration with cloud-native workload management.

It is particularly useful in environments where jobs are frequently rescheduled while the underlying high-performance network infrastructure remains operational.

Dividing congestion control between local and global components

#

RDMA congestion control must respond quickly enough to prevent network overload without sacrificing throughput.

DistDPU assigns different responsibilities to the OM and EM.

The OM manages global configuration, selects congestion-control algorithms such as DCQCN, and distributes parameters across execution modules.

The EM handles fast local feedback in the forwarding path, where delays associated with centralized decision-making would be undesirable.

This separation allows the architecture to retain global policy coordination without requiring every congestion event to pass through the central management processor.

It also helps preserve the low-latency behavior required by distributed training and inference.

VXLAN and high-performance NAT modes

#

DistDPU supports different packet-processing paths depending on the isolation and performance requirements of a deployment.

In shared multi-tenant environments, VXLAN encapsulation provides an overlay mechanism for separating tenant networks. Encapsulation allows multiple logical networks to share physical infrastructure while maintaining distinct network identities.

For dedicated single-tenant deployments, the architecture can use one-to-one network address translation instead of VXLAN encapsulation.

NAT mode rewrites packet addresses through a table-driven hardware pipeline, avoiding some of the overhead associated with tunnel encapsulation.

This is a deployment trade-off rather than a universal replacement for VXLAN. Overlay networking remains useful when flexible tenant isolation is required, while the NAT path can be advantageous when the hardware and allocation model permit a simpler forwarding configuration.

🔗 M2M Fabric: Coordinating Multiple Execution Modules

#

Disaggregating the DPU creates a new management requirement: the OM must configure and monitor multiple physically separate EMs while preserving the appearance of a unified device.

DistDPU introduces the M2M Fabric as an internal communication mechanism for this purpose.

Internal configuration and monitoring channel

#

The M2M Fabric provides interfaces for operations including:

• Distributing device configurations and forwarding rules.
• Querying the status of individual EMs.
• Collecting operational telemetry and fault reports.
• Coordinating lifecycle and management operations.
• Exchanging remote procedure call (RPC)-style messages between modules.

The communication mechanism can use PCIe, in-band Ethernet, or supported inter-board links, allowing the architecture to accommodate different hardware layouts.

A local EM can also act as a relay, forwarding management traffic from the OM to remote EMs across supported PCIe topologies. This reduces the need for the OM to maintain a separate direct connection to every execution module.

Isolating management traffic from tenant workloads

#

Security isolation is essential because the M2M Fabric carries privileged management messages.

Tenant-side processes must not be able to access or manipulate these messages. DistDPU therefore keeps the internal management mechanism isolated from ordinary tenant data paths.

This separation supports the broader goal of preserving DPU-style isolation while allowing the forwarding hardware to be expanded independently.

Management latency

#

The reported measurements show a latency of approximately 2.2 microseconds for a 4-byte management write and 4.9 microseconds for a read.

These figures characterize the measured management operations, not the latency of ordinary RDMA packet transfers.

Management traffic is relatively infrequent compared with the data path, so the architecture does not require every packet to traverse the M2M Fabric. Instead, forwarding rules are installed ahead of time and executed locally in the EM hardware.

The result is a control path designed for configuration and management rather than packet-by-packet forwarding.

🛡️ Reliability and Fault Isolation

#

A disaggregated architecture must prevent failures in one component from destabilizing the complete networking system.

DistDPU separates management responsibilities from forwarding operations to contain faults and preserve active traffic whenever possible.

Execution Module failures

#

EM failures can result from component faults, optical modules, fibers, or other physical connectivity problems.

The architecture uses hardware-level fault handling and supported dual-port failover mechanisms to reroute traffic when a path becomes unavailable.

Because the cloud platform sees a unified logical DPU, the management layer can isolate the affected module rather than treating every device as an entirely independent networking system.

The practical recovery behavior still depends on network topology, redundant paths, and the specific hardware failure.

Orchestration Module process failures

#

The OM is responsible for configuration and management, but it should not need to participate in every packet-forwarding operation after the forwarding tables have been installed.

DistDPU includes a watchdog that can restart the OM management process if it crashes.

Meanwhile, EMs continue using the forwarding rules already loaded into their hardware tables. Existing traffic can therefore continue without requiring the management process to be active for every operation.

This design minimizes the immediate impact of a control-process failure, although operations requiring new policies, configuration updates, or recovery actions may remain unavailable until management is restored.

The broader engineering principle is that failure of the control plane should not automatically imply failure of the data plane.

📊 Production Evaluation: Cost, Power, and Operational Efficiency

#

The research team evaluated DistDPU against two alternative deployment approaches:

1. Multi-BF3: Stacking multiple full NVIDIA BlueField-3 DPUs.
2. Multi-CX7: Stacking NVIDIA ConnectX-7 network interface cards.

These comparisons address different trade-offs. Multi-BF3 provides full DPU functionality but duplicates management resources. Multi-CX7 offers high-performance networking at lower hardware complexity, but does not provide the same complete DPU isolation and control-plane capabilities by itself.

Hardware cost and power consumption

#

DistDPU’s most significant reported advantage is the reduction in hardware required to scale network bandwidth.

The figures are the research team’s reported results and represent maximum measured or reported improvements, not guaranteed savings for every configuration.

The mechanism behind the reduction is straightforward: the architecture avoids repeatedly purchasing a complete SoC, memory subsystem, and management software stack when the actual requirement is additional forwarding bandwidth.

The operational gains follow a similar logic. Managing one logical DPU with several subordinate execution modules can reduce the number of independently managed entities, simplifying configuration, monitoring, and coordinated upgrades.

Actual savings will depend on hardware pricing, the number of EMs required, topology, software integration, and the scale of the deployment.

Orchestration CPU utilization remains low

#

Even at Tbps-scale aggregate bandwidth, the research reports that average CPU utilization on the OM remained below 15%.

This result supports the central architectural motivation: the control-plane processing required for the tested workloads remained modest relative to the data-plane capacity provided by the execution modules.

The result should not be interpreted as a universal upper bound for OM utilization. Workloads with frequent policy changes, unusually high connection churn, or different virtualization requirements could produce a different control-plane profile.

Nevertheless, the production measurements provide evidence that separating the two scaling dimensions can improve resource utilization in large AI cloud deployments.

Aggregate bandwidth reaches 3.6 Tbps per server

#

DistDPU supports up to 3.6 Tbps of aggregate bandwidth per server, according to the reported deployment results.

This is an aggregate figure across the server’s network interfaces and execution modules, rather than a claim that one individual link operates at 3.6 Tbps.

The architecture increases that aggregate capacity by adding forwarding resources without requiring the same proportional increase in full-featured DPUs.

For GPU clusters, this model can be attractive when adding compute nodes and accelerators also requires substantially more networking capacity, but does not create an equivalent increase in network-management work.

🚀 RDMA Throughput and AI Workload Performance

#

A cost-efficient networking architecture is only useful if its virtualization and forwarding mechanisms do not significantly degrade application performance.

DistDPU’s evaluation therefore includes low-level RDMA measurements and end-to-end AI training and inference workloads.

RDMA throughput remains close to commercial hardware

#

Across the reported tests, DistDPU’s RDMA performance remained within approximately 0.4%–2.8% of the commercial hardware baselines.

These results indicate that the additional abstraction introduced by disaggregation can be implemented without imposing a substantial throughput penalty in the evaluated configurations.

The architecture also benefits from its ability to select different forwarding modes for different deployment requirements.

For large packet sizes, the reported NAT mode delivered an additional 1.2%–7.8% throughput improvement over VXLAN mode.

The improvement is consistent with removing tunnel-processing overhead where overlay encapsulation is not required. It should not be interpreted as evidence that NAT is always faster for every packet size, traffic pattern, or network configuration.

For additional background on the data-path technology, KAD provides an explanation of RDMA and its role in high-performance computing and AI infrastructure.

Large-model training results

#

The researchers evaluated AI training workloads involving Llama 2 7B and Mixtral 8x7B.

For dense-model training, DistDPU delivered performance comparable to the commercial baselines. In the evaluated mixture-of-experts (MoE) workload, the study reported an improvement of up to approximately 1.26%.

This result matters because distributed training depends on communication patterns as well as raw GPU compute throughput. AllReduce and other collective operations can become bottlenecks when gradients and intermediate data must move between many accelerators.

A networking system that provides additional bandwidth while maintaining RDMA performance can help reduce communication-related overhead without forcing the training framework to adopt a new protocol.

The reported difference is modest, but it demonstrates that the disaggregated architecture did not introduce a large training-performance penalty in the tested workloads.

DeepSeek-R1 inference results

#

The evaluation also examined DeepSeek-R1 inference using metrics such as:

• Time to First Token (TTFT): The delay between submitting a request and receiving the first generated token.
• Time Per Output Token (TPOT): The average time required to generate subsequent output tokens.

Across the reported tests, both metrics differed by less than 1% from the commercial hardware baselines.

These results suggest that the network architecture’s additional management abstraction did not materially degrade the evaluated inference performance.

However, end-to-end latency also depends on model execution, GPU scheduling, request batching, memory access, and service architecture. The results should therefore be understood as measurements under the paper’s experimental conditions rather than a general guarantee for every inference workload.

🧠 Practical Lessons for AI Cloud Infrastructure

#

DistDPU illustrates several broader principles for designing networks that serve large GPU clusters.

GPU and CPU resource allocation must evolve together

#

AI workloads are often dominated by GPU computation, but the servers hosting those GPUs still require networking, security enforcement, orchestration, and other infrastructure services.

In some deployments, host CPU resources may remain underutilized while infrastructure requirements continue to grow.

Separating network control and forwarding resources offers one way to improve efficiency. It allows operators to reserve only the management capacity needed for orchestration and allocate more resources to the data path when bandwidth becomes the primary constraint.

The research team also discusses opportunities to co-locate general-purpose workloads with AI infrastructure. Recovering otherwise unused server capacity could improve utilization and revenue, but doing so safely requires fine-grained isolation and careful control of contention involving PCIe, forwarding tables, memory, and network bandwidth.

Failure handling must be designed into the architecture

#

A disaggregated system creates more components and interconnections, so its reliability model must account for failures across those boundaries.

DistDPU separates fault domains where possible: an individual EM can be isolated, while the OM management process can restart without automatically interrupting forwarding through already configured hardware tables.

This illustrates a useful design principle for infrastructure systems: critical data paths should not depend unnecessarily on continuous operation of the processes responsible for configuring them.

Monitoring, recovery, and failover mechanisms must still be validated against the physical topology and the failure modes of the actual deployment.

RDMA compatibility lowers adoption barriers

#

AI infrastructure depends on established communication interfaces and mature software stacks. Requiring applications to adopt a proprietary transport or change their distributed training frameworks can make an otherwise attractive networking architecture difficult to deploy.

DistDPU maintains compatibility with standard RDMA Reliable Connection (RC) semantics.

Its EM hardware handles multipath and potentially out-of-order completion behavior internally while preserving the ordered completion semantics expected by applications.

This allows the network infrastructure to change without requiring corresponding modifications to upper-layer AI software.

For cloud providers, compatibility is a practical requirement, not merely a convenience. It reduces migration costs and allows existing AI workloads to use the new architecture through familiar network interfaces.

✅ Conclusion: Disaggregating the DPU Changes How AI Networks Scale

#

DistDPU addresses a structural inefficiency in conventional cloud networking: network bandwidth may need to grow much faster than the computing resources required to manage it.

By separating the Orchestration Module from lightweight Execution Modules, the architecture allows control-plane resources and data-plane bandwidth to scale independently. The OM centralizes policy, virtualization, configuration, and monitoring, while EMs perform high-speed Ethernet and RDMA forwarding in hardware.

Its reported production results are promising. DistDPU supports more than 10,000 GPUs, reaches up to 3.6 Tbps of aggregate bandwidth per server, and reduces single-node hardware costs and power consumption by up to 77.1% and 59.3%, respectively, compared with stacking full-featured BlueField-3 DPUs.

The measured RDMA, training, and inference results also indicate that the architecture can preserve performance close to commercial hardware baselines while reducing infrastructure overhead.

The broader lesson is that scaling AI infrastructure does not always require replicating every component of an existing system. When control-plane requirements grow more slowly than data-plane demands, separating these resource dimensions can improve efficiency, simplify management, and reduce operating costs.

For AI cloud operators, DistDPU provides a concrete example of how hardware disaggregation can turn network bandwidth expansion from a full-device scaling problem into a more targeted forwarding-capacity problem.

Its long-term value will depend on deployment economics, fault recovery, hardware integration, and the range of workloads that can benefit from the architecture. Nevertheless, the research presents a compelling direction for building high-performance AI clouds without allowing network infrastructure costs to grow in lockstep with bandwidth.