NJU × Huawei UBEP: Reengineering MoE Expert Parallelism
UBEP: Re-architecting Expert Parallelism Communication Library for Production Superpods
Mixture-of-Experts (MoE) has become a mainstream architecture for scaling large language models without making every token pay the compute cost of the full parameter set. Models such as DeepSeek-R1, Qwen3, and GLM use sparse expert activation to increase model capacity while keeping per-token computation manageable.
That scaling strategy introduces a different bottleneck: communication.
In conventional dense-model data parallelism, communication is dominated by structured collective operations such as AllReduce, with relatively predictable synchronization behavior. Expert Parallelism (EP) has a fundamentally different communication pattern. Experts are distributed across accelerators, so each token must be dispatched to the device hosting its selected expert and later gathered during the Combine phase.
The result is a sparse, irregular All-to-All exchange across accelerators.
According to the evaluation presented in the paper, MoE communication can consume roughly 47% of inference execution time. As accelerator interconnects become dramatically faster, this problem becomes even more pronounced: the time required to move tokens decreases, exposing synchronization, scheduling, and software coordination overheads that older communication libraries were designed to tolerate.
At SIGCOMM 2026, a joint research team from Nanjing University and Huawei introduced UBEP (Unified-Bus Expert Parallelism), a redesigned MoE expert-parallel communication library targeting large-scale superpods. Evaluated on Huawei’s CloudMatrix384 (CM384), UBEP reduced All-to-All communication latency by up to 52.4% and improved end-to-end MoE inference TPOT (Time Per Output Token) by up to 11.1%.
The key idea is not simply to optimize individual communication kernels. UBEP restructures the entire communication path around the characteristics of high-bandwidth, low-latency unified interconnects.
🚀 Why Superpods Expose the Limits of Legacy MoE Communication #
Understanding UBEP requires looking at the hardware environment it targets.
From Conventional Clusters to Unified Superpods #
Traditional AI clusters typically combine high-speed intra-node links such as NVLink with scale-out networking technologies such as InfiniBand or RoCE. These networks provide substantial bandwidth, but their relatively high communication latency can hide software overhead.
When data transfers themselves take a significant amount of time, synchronization and scheduling overhead may represent only a small fraction of the overall communication cost.
Superpods change that equation.
Architectures such as NVIDIA NVL72/NVL576 and Huawei CloudMatrix384 connect large numbers of accelerators through high-bandwidth Scale-Up fabrics. CM384 uses Huawei’s Unified-Bus (UB) architecture to create a much tighter communication domain across NPUs.
The paper reports UB bandwidth approaching 400 GB/s, substantially higher than the approximately 50 GB/s class of traditional RoCE links, with communication latency reaching the hundreds-of-nanoseconds range.
More importantly, CM384 provides a Unified Global Address Space (UGAS). An accelerator can directly perform load/store operations against remote accelerator memory rather than relying exclusively on explicit message-passing semantics.
Conceptually, the architecture changes from:
accelerator → NIC → network → NIC → accelerator
to a shared high-speed memory-access domain in which accelerators can directly interact with remote memory.
This dramatically accelerates data movement, but it also changes where the bottlenecks reside.
Once token transfers become fast enough, software synchronization and scheduling operations that were previously hidden behind communication latency become first-order performance costs.
⚠️ Three Bottlenecks in Existing EP Communication Libraries #
The UBEP paper identifies three major problems in conventional MoE expert-parallel communication implementations.
BSP Serialization #
Existing MoE communication libraries, including systems such as DeepEP and CANN EP, largely follow a Bulk Synchronous Parallel (BSP) execution model.
Under BSP, compute units execute a sequence of phases and synchronize at global barriers before advancing. A typical communication pipeline may therefore resemble:
- Send tokens.
- Send token counts.
- Calculate offsets.
- Reorder tokens.
- Synchronize.
- Continue to the next phase.
This approach creates unnecessary serialization.
A faster accelerator or AIV core can finish its work early but must wait for slower participants to reach the same synchronization point. On a high-speed superpod, token movement can complete quickly enough that metadata processing and synchronization become visible bottlenecks.
The result is reduced utilization of both accelerator compute resources and the underlying interconnect.
The Synchronization Tax #
The second problem is the growing cost of synchronization itself.
Traditional communication designs separate payload data from control information. Tokens are transmitted as data, while completion flags, barriers, and other signals are handled through separate control operations.
That separation made sense when data movement was comparatively expensive.
On a superpod, however, the balance changes. If a token transfer takes only a small fraction of a microsecond, the additional operations required to signal completion, launch kernels, synchronize participants, and poll status can become a substantial percentage of total latency.
The authors describe this overhead as the Synchronization Tax.
Their measurements show that global SyncAll synchronization can consume as much as 15% of total execution time, with the impact increasing as system scale grows.
Topology-Agnostic Scheduling #
The third bottleneck is the assumption that the unified memory system behaves like a uniform network.
Traditional schedulers commonly balance work according to token counts. If two AIV cores process the same number of tokens, they are assumed to have approximately equivalent completion times.
That assumption breaks down on hierarchical superpod networks.
CM384 contains multiple switch tiers, meaning that remote memory accesses can have different hop counts and therefore different latency characteristics. The paper reports that two-hop access latency can be 11.5× higher than local access.
A scheduler that assigns equal token counts without considering network distance can therefore produce significant tail latency.
The software sees a unified address space, while the physical network remains hierarchical and non-uniform.
UBEP is designed around this distinction.
🧩 UBEP’s Three Core Innovations #
UBEP addresses the three bottlenecks through three complementary mechanisms:
- Kernel Decomposition to remove unnecessary execution serialization.
- Hierarchical Token-Level Scheduling to account for network topology.
- Data-as-Flag to combine synchronization and data movement.
Together, these mechanisms turn the communication stack from a globally synchronized pipeline into a more asynchronous, topology-aware system.
Kernel Decomposition #
Under a conventional BSP implementation, AIV cores may execute communication and metadata operations sequentially:
Send Tokens → Send Counts → Calculate Offsets → Reorder Tokens
UBEP decomposes this pipeline and assigns specialized roles to different AIV cores.
Most AIV cores focus on token data movement, while a smaller subset handles metadata operations such as token counting, prefix sums, and offset calculation.
This enables metadata processing to proceed concurrently with token transmission.
UBEP also replaces global SyncAll barriers with asynchronous peer-to-peer signaling. Once metadata is calculated, the result is written directly into global shared memory. Other AIV cores can poll the relevant memory location and begin token reordering as soon as the required data becomes available.
The communication pipeline therefore changes from a globally synchronized sequence into overlapping stages.
Adaptive AIV Allocation #
UBEP also provides an analytical model for selecting how many AIV cores should be dedicated to metadata processing.
The required metadata-processing capacity is related to the total number of experts and inversely related to the product of:
batch_sizeTop-k
As batch size and token-processing pressure increase, more AIV resources can be dedicated to moving token data, while the metadata-processing allocation can be adjusted accordingly.
The important design principle is that AIV allocation should follow the communication workload rather than remain statically partitioned.
Hierarchical Token-Level Scheduling #
UBEP’s second innovation makes token scheduling explicitly topology-aware.
The scheduling problem can be formulated as assigning m tokens across n AIV cores while minimizing the maximum completion time.
Token count alone is insufficient. The scheduler must simultaneously consider:
- Total token volume assigned to each AIV.
- The proportion of low-latency and high-latency traffic.
- Network hop count.
- Potential contention on remote paths.
The paper shows that the exact optimization problem is NP-hard, making exact online optimization unsuitable for inference workloads that require scheduling decisions within the microsecond range.
UBEP therefore introduces a heuristic called Latency Homogenization.
Latency Homogenization #
The scheduler first classifies tokens according to their network hop count. It then distributes tokens so that individual AIVs receive a balanced mixture of different latency classes.
The objective is not simply to equalize token counts. Instead, UBEP attempts to equalize the expected communication cost of each AIV.
This reduces the likelihood that one AIV becomes a straggler because it happens to receive a disproportionate amount of two-hop traffic.
UBEP further accelerates the mapping operation using a hardware-oriented Quaternary Search SIMD algorithm, reducing the lookup process to roughly four to five steps.
The paper reports that the resulting scheduling overhead remains below 1 μs, making the topology-aware optimization practical for inference.
Ablation Results #
The scheduling ablation demonstrates why topology awareness matters.
| Configuration | Peak AIV Latency |
|---|---|
| CANN-EP baseline | 62.2 μs |
| UBEP without topology-aware mapping | 48.1 μs |
| Full UBEP | 43.5 μs |
A token-count-only scheduler improves load balance, but significant two-hop traffic can still concentrate on individual AIVs.
Full UBEP balances both token volume and network-hop distribution, reducing peak latency to 43.5 μs.
The approach can also be generalized to superpods with more than two switch tiers as long as measurable latency differences exist between different hop classes.
Data-as-Flag #
UBEP’s third innovation addresses synchronization at the memory-operation level.
Traditional communication follows a pattern similar to:
write data → send completion flag → receiver polls flag → consume data
This separates payload movement from readiness signaling.
On CM384, however, the hardware supports 512-byte atomic writes. A complete 512-byte write becomes visible to the remote side atomically rather than exposing partially written fragments.
UBEP uses this property to fuse data and synchronization.
The paper evaluates three Data-as-Flag mechanisms:
TFF: Token-Flag Fusion #
TFF divides each 512-byte block into:
- 32-byte synchronization metadata.
- 480-byte token payload.
The sender atomically writes the complete 512-byte block. When the receiver observes the updated flag, it can infer that the corresponding payload is also available.
The primary trade-off is a small amount of payload bandwidth consumed by the embedded flag.
DC: Data-for-Checksum #
Data-for-Checksum removes per-block embedded flags.
Instead, the sender writes the data and subsequently transmits a consolidated checksum packet. The receiver waits for the checksum before consuming the associated data.
This approach minimizes bandwidth overhead but sacrifices some fine-grained pipelining because the receiver must wait for completion of the corresponding data batch.
SP: Sentinel Polling #
Sentinel Polling takes a different approach.
The receiver initializes the destination buffer with a designated sentinel value. The sender then writes the raw payload without modifying it.
The receiver polls memory and detects a value different from the sentinel to determine that data has arrived.
This avoids both separate control messages and embedded metadata, allowing bandwidth utilization to approach 100%.
The challenge is correctness: a legitimate payload could theoretically contain the sentinel value.
UBEP addresses this by selecting a bit pattern that cannot occur in the relevant MoE activation representation. Under the paper’s assumptions, the resulting collision probability is approximately 2⁻²⁵⁶.
Among the evaluated approaches, SP provides the best overall trade-off between bandwidth efficiency and pipeline granularity.
Data-as-Flag Performance #
The three Data-as-Flag variants reduce latency by approximately 31% to 57.1% compared with the traditional Stop-and-Wait (SW) mechanism.
The broader lesson is important: once remote memory access becomes sufficiently fast, synchronization metadata itself becomes part of the critical path. Eliminating that extra control plane can therefore produce substantial gains.
📊 Experimental Evaluation on Huawei CloudMatrix384 #
UBEP was evaluated on Huawei’s CloudMatrix384 superpod using 16 servers and 256 NPU dies.
The experiments covered several mainstream MoE workloads, including:
- Qwen3-30B
- GLM-4.7
- DeepSeek-R1
- DeepSeek-V3.2
The results show improvements at both the communication-operator and end-to-end inference levels.
Communication-Level Performance #
UBEP achieved:
- Up to 52.4% lower All-to-All communication latency.
- 35.3%–40.8% higher effective bandwidth.
- Average speedups of 46.4% across different batch sizes.
- Average speedups of 43.9% across different NPU counts.
These results demonstrate that the largest gains come from eliminating software-induced serialization and synchronization rather than simply increasing raw network bandwidth.
End-to-End MoE Inference #
At the full inference level, UBEP improved TPOT by up to 11.1%.
At first glance, a 52.4% reduction in the communication operator might appear inconsistent with an 11.1% end-to-end improvement. The difference comes from the composition of the inference pipeline.
MoE communication is only one component of total execution. The remaining time includes:
- FFN computation.
- Attention.
- Framework scheduling.
- Kernel launches.
- Memory movement.
- Other synchronization and execution overhead.
The paper’s analysis indicates that MoE communication can account for roughly half of latency observed in the relevant token-processing path while representing only around 20% of total hardware execution time.
Consequently, optimizing communication alone cannot produce a proportional end-to-end speedup.
This also points toward the next class of optimization opportunities: applying the same dependency-driven, asynchronous execution principles to other stages of the inference pipeline.
🔬 What UBEP Changes About MoE Communication Design #
The significance of UBEP extends beyond its measured speedups.
The system illustrates a broader shift in AI infrastructure design.
Hardware Is No Longer Hiding Software Inefficiency #
Older communication stacks were designed around relatively expensive data movement. Synchronization and metadata operations could often be hidden behind network latency.
Superpods invert that relationship.
When the underlying interconnect becomes fast enough, software operations that previously appeared insignificant become dominant:
- Global barriers.
- Kernel launch overhead.
- Metadata processing.
- Control-message exchanges.
- Topology-unaware scheduling.
- Polling and readiness detection.
In other words, faster hardware exposes software architectural debt.
Unified Memory Does Not Mean Uniform Latency #
UGAS provides a unified programming and addressing model, but it does not eliminate the physical topology of the underlying network.
This distinction is critical for future distributed accelerator systems.
A scheduler that treats every remote memory access as equivalent can be functionally correct while still being performance-inefficient.
UBEP demonstrates that high-performance software needs to expose enough of the physical topology to make intelligent scheduling decisions, even when the programming model presents a unified address space.
Synchronization Becomes a First-Class Data-Path Problem #
Data-as-Flag is perhaps the clearest example of hardware/software co-design in UBEP.
Instead of treating synchronization as a separate control-plane operation, UBEP embeds readiness semantics into the data path itself.
This reduces the number of communication primitives required to complete an operation and allows memory visibility to serve as the synchronization mechanism.
As accelerator interconnects continue moving toward lower latency and stronger remote-memory semantics, similar techniques are likely to become increasingly relevant beyond MoE.
🏁 Conclusion #
The performance bottleneck in large-scale MoE systems is increasingly shifting away from raw interconnect bandwidth and toward the software stack that orchestrates communication.
Traditional Expert Parallel Communication Libraries were largely designed around the characteristics of conventional clusters. Their BSP synchronization, separate control paths, and topology-agnostic scheduling assumptions become increasingly inefficient when deployed on high-bandwidth, multi-tier superpods.
UBEP addresses these limitations at three levels:
- Kernel Decomposition removes unnecessary global serialization and overlaps metadata processing with token movement.
- Hierarchical Token-Level Scheduling accounts for non-uniform network latency and reduces AIV stragglers.
- Data-as-Flag fuses synchronization with memory writes to eliminate unnecessary control traffic.
On Huawei CloudMatrix384, these changes translate into up to 52.4% lower All-to-All latency and up to 11.1% better MoE inference TPOT.
The broader takeaway is that accelerator performance improvements increasingly require communication software to be redesigned alongside the hardware. As superpods scale to hundreds or thousands of accelerators, low-level communication libraries, schedulers, synchronization primitives, and execution runtimes will need to become topology-aware, asynchronous, and tightly coupled to the capabilities of the underlying interconnect.
UBEP is a concrete example of that transition: rather than simply making the network faster, it redesigns the software around the assumption that the network is already fast.