STORM Explained: NIC-Level RDMA Scheduling at SIGCOMM 2026

Remote Direct Memory Access (RDMA) has become a fundamental technology in modern data centers. By allowing one machine to access another machine’s memory with minimal CPU involvement, RDMA delivers microsecond-level latency and high throughput. It is widely used in key-value storage, cloud infrastructure, distributed storage systems, high-performance computing (HPC), and large-scale AI training.

However, as RDMA traffic increases, a fundamental networking question becomes increasingly important: when multiple requests compete for the same bandwidth, which one should the network transmit first?

Traditional network scheduling often relies on fair sharing, distributing available bandwidth among competing flows. Although straightforward, this approach does not account for the different completion requirements and dependency relationships of modern distributed workloads. A short request nearing completion, for example, may deserve priority over a large transfer. Conversely, a large transmission blocking dozens of subsequent operations may be more important than several unrelated short messages.

The paper STORM: Enabling Traffic Scheduling for RDMA, presented at ACM SIGCOMM 2026, addresses this problem through a scheduling mechanism implemented directly inside the RDMA Network Interface Card (RNIC).

Jointly developed by researchers from the University of Cambridge, Microsoft Research, Politecnico di Milano, and Queen Mary University of London, STORM introduces a practical scheduling policy based on two pieces of information already available to the NIC:

• Remaining request size: How many bytes still need to be transmitted?
• QP backlog: How many logical transactions are waiting behind the current request?

Its central insight is that scheduling should consider not only how close a request is to completion, but also how much subsequent work is blocked by it.

By translating these two signals into a small number of network priority levels, STORM attempts to accelerate the requests that contribute most directly to overall application progress—without requiring application-level scheduling hints or centralized coordination.

🚀 Why Traditional RDMA Fair Sharing Is Not Enough

#

RDMA provides efficient data movement, but efficient transport does not automatically guarantee efficient workload completion.

In many production data centers, RDMA traffic is assigned to a service-level priority class and then handled using relatively simple fair sharing. This treats competing transfers largely as independent flows, even when their application-level dependencies are very different.

Consider two requests competing for a network link:

• Request A has only a small amount of data remaining.
• Request B is a much larger transfer.

With equal bandwidth sharing, both requests continue transmitting simultaneously. Request A cannot finish until it receives enough bandwidth, potentially delaying the completion of a small operation that could otherwise have released resources quickly.

For independent requests, this behavior is inefficient.

However, simply prioritizing the smallest request is not universally correct either. In a distributed storage system or an AI training cluster, a large transmission may sit at the head of a queue and prevent many subsequent operations from progressing.

The scheduling problem therefore has two dimensions:

1. Completion efficiency: Prioritize requests with little remaining data when requests are otherwise independent.
2. Dependency awareness: Accelerate requests that are blocking substantial amounts of subsequent work.

Traditional Shortest Remaining Processing Time (SRPT) scheduling primarily addresses the first dimension. STORM extends this idea by introducing queue backlog as a second scheduling signal.

The result is a scheduling mechanism designed for both independent cloud traffic and dependency-heavy distributed workloads.

🧭 Three RDMA Workload Categories With Different Scheduling Requirements

#

The STORM paper considers three major workload categories. Each exposes a different reason why conventional fair sharing or packet-size-based scheduling can perform poorly.

1. Request-Response Cloud Workloads

#

Typical examples include:

• Key-value storage services
• Remote procedure calls (RPCs)
• Cloud application backends
• Distributed caching systems

These workloads generally generate large numbers of independent requests with widely varying transfer sizes.

Suppose a 4 KB request and a 4 MB request arrive simultaneously.

Under equal bandwidth sharing, the small request must compete with the large transfer even though it could finish quickly with a dedicated portion of the link.

Prioritizing the smaller request allows it to complete earlier. The large transfer experiences a relatively small delay in comparison, while the application benefits from faster completion of the short operation.

This is the fundamental motivation behind Shortest Remaining Processing Time (SRPT) scheduling.

Importantly, SRPT considers the remaining processing requirement rather than the original request size. A request that started as a large transfer may deserve priority once it approaches completion.

For independent request-response traffic, this strategy can substantially reduce average flow completion time and tail latency.

2. Storage Services: Fan-In and Fan-Out I/O

#

Distributed storage systems frequently split one logical application request across multiple storage servers.

A client may need to retrieve data from several backends simultaneously. Completion occurs only after all required responses have arrived.

This creates dependency relationships that cannot be captured by packet size alone.

Consider a backend QP containing several queued transactions. The request currently at the head of the queue prevents subsequent transactions from progressing. Even when those subsequent operations are relatively small, they cannot move forward until the blocking transaction completes.

Meanwhile, accelerating unrelated short responses from other servers may provide little benefit to the application because the missing data remains unavailable.

The scheduling objective is therefore different from ordinary SRPT:

Prioritize transmissions that unblock subsequent work, rather than blindly prioritizing the smallest transfer.

By observing QP backlog, STORM can identify requests that potentially prevent more operations from progressing.

3. Distributed AI Training and Collective Communication

#

Large-scale AI training introduces even more complicated communication dependencies.

Distributed training frameworks such as NCCL use collective communication operations, including:

• AllReduce
• ReduceScatter
• AllGather
• Broadcast

These operations coordinate communication across multiple GPUs and machines.

The completion time of a collective operation is often determined by its slowest participant. A single delayed node can become a straggler, preventing the collective from completing and delaying subsequent computation and communication.

This creates pipeline bubbles in distributed training.

In such environments, prioritizing the smallest network transfer is not necessarily the best strategy. A larger transmission may belong to a critical dependency chain and be responsible for blocking several subsequent operations.

STORM attempts to identify these situations through QP backlog.

When a transmission has substantial queued work behind it, completing it can release more downstream operations and help eliminate communication stalls.

The important distinction is that STORM does not attempt to understand the complete application dependency graph. Instead, it uses locally observable NIC metadata as a practical approximation of dependency importance.

🧠 The Two Critical Scheduling Signals Inside an RDMA NIC

#

STORM’s most important design decision is to avoid introducing new application-level metadata requirements.

An RNIC already has information about the work requests submitted to it. The researchers use two existing signals to build a lightweight scheduling policy.

1. Remaining Request Size

#

When an application submits an RDMA work request, it specifies the amount of data to transfer.

The NIC therefore knows the original request size. By tracking how many bytes have already been transmitted, it can calculate the remaining transmission size.

Conceptually:

[ \text{Remaining Size}

#

\text{Original Size}

#

\text{Transmitted Bytes} ]

This provides the information required for SRPT-style scheduling.

A request that originally contained a large amount of data may become a high-priority candidate after most of its payload has already been transmitted.

This is more accurate than scheduling according to the original message size alone.

2. Queue Pair Backlog

#

An RDMA Queue Pair (QP) contains a Send Queue and a Receive Queue. For STORM’s scheduling mechanism, the important information is the queued logical work associated with the transmission.

When multiple logical transactions are waiting behind the current head-of-line transaction, the head transaction potentially blocks their progress.

The paper refers to this queued work as backlog.

A deeper backlog means that completing the current transaction may release more subsequent work.

It is important to distinguish this metric from ordinary network congestion measurements.

STORM’s backlog does not represent:

• The number of packets waiting in a switch buffer.
• The amount of data queued in bytes.
• The number of packets currently in flight.
• The total congestion experienced by the network.

Instead, it represents the number of logical transactions queued behind the current transaction within a QP.

This provides a local indication of how much subsequent work may be waiting for the head-of-line request to complete.

3. Combining Backlog and Remaining Size

#

STORM combines these two signals into a straightforward urgency ordering:

1. A QP with a deeper backlog receives higher scheduling priority.
2. When backlog values are equal, the request with fewer remaining bytes receives higher priority.

The resulting policy can be summarized as follows:

This policy preserves the advantages of SRPT for independent requests while providing additional priority to requests that block subsequent operations.

The key is that both inputs are available to the RNIC. The scheduler does not need a separate global scheduling service or continuous application-level reporting.

4. Logical Transaction Grouping

#

There is an important implementation detail.

Software libraries and communication frameworks frequently divide one large logical transaction into multiple smaller work requests or fragments before submitting them to the NIC.

If STORM simply counts every fragment as an independent transaction, it may significantly overestimate the actual backlog.

For example, one logical 4 MB transfer split into dozens of fragments should not necessarily be interpreted as dozens of independent application transactions.

To address this problem, STORM introduces a task-grouping mechanism.

Multiple fragmented work requests belonging to the same logical transaction are grouped into one logical unit. Backlog and remaining-size calculations operate on the logical transaction rather than treating every fragment as an independent scheduling candidate.

The grouping mechanism works best when the upper layer provides a common transaction identifier.

When such an identifier is unavailable, STORM can use conservative temporal grouping as a fallback. This reduces scheduling precision slightly but avoids requiring every application to expose additional metadata.

This distinction is essential: STORM schedules logical work rather than merely prioritizing individual packets or raw work-request fragments.

⚙️ How STORM Works: Sender-Side Priority Marking and Receiver-Side Scheduling

#

Turning a continuous urgency ranking into an actual network scheduling mechanism introduces two important challenges.

First, network switches have limited priority resources. A scheduler may be able to distinguish hundreds of requests, but the underlying network generally provides only a small number of priority queues.

Second, the sender and receiver have different visibility.

The sender knows its own request sizes and QP backlog immediately. However, it cannot directly observe contention from other senders targeting the same receiver.

The receiver can observe traffic arriving from multiple senders, but waiting for feedback requires at least one network round-trip time.

STORM addresses these limitations by dividing scheduling responsibilities between the sender and receiver.

Its primary design reference is the Bandwidth-Delay Product (BDP).

BDP represents the amount of data that can be in flight over a link during one round-trip time. It provides a useful boundary between transfers that can be completed immediately and larger transfers that require receiver-side coordination.

1. Sender-Side Scheduling: Assigning Packet Priorities

#

STORM’s default configuration uses three priority levels:

• P1: Highest priority
• P2: Intermediate priority
• P3: Lowest priority

Smaller priority numbers represent higher scheduling priority.

The sender makes an initial decision based on request size and backlog.

Short messages: no larger than one BDP

For a message whose total size is less than or equal to one BDP, the sender assigns P1 to the entire message.

This allows short transfers to complete without waiting for receiver-side scheduling feedback.

Large messages: larger than one BDP

For messages exceeding one BDP, the sender immediately transmits the first BDP of data.

The initial priority depends on whether the current QP has queued work:

This distinction ensures that large requests blocking subsequent work receive preferential treatment over ordinary large transfers.

After transmitting the initial BDP, a large message cannot continue indefinitely at its initial priority.

Further transmission becomes subject to receiver-side scheduling decisions.

This division allows STORM to avoid waiting for a full feedback round trip before starting a transfer while still coordinating long-running transmissions.

2. Receiver-Side Scheduling: Managing an Active QP Set

#

The receiver coordinates long transfers using a small active QP set.

The purpose is to avoid allowing too many large transfers to consume network resources simultaneously.

In the default three-priority configuration, STORM maintains two scheduling slots:

Only QPs admitted to the active set can continue transmitting data beyond their initially authorized portion.

The receiver selects candidates according to the same urgency principle:

1. Prefer the QP with the deeper backlog.
2. Break ties using the smaller remaining request size.

Once a QP receives permission to continue, the receiver authorizes a relatively large transmission window.

This is a form of coarse-grained transmission authorization rather than per-packet rate control.

The distinction matters because STORM does not attempt to replace the underlying congestion-control algorithm.

The receiver determines which QPs are eligible to continue transmitting, while congestion control continues to regulate the actual sending rate.

3. Why BDP-Based Scheduling Matters

#

Waiting for receiver feedback before transmitting any data would introduce unnecessary startup latency.

For a large message, the initial BDP allows the sender to begin transmitting immediately.

Only after that initial transmission does receiver-side coordination become necessary.

This design balances two competing requirements:

• Keep the network pipeline occupied.
• Prevent too many long transfers from competing simultaneously.

It also allows the receiver to make better decisions because it can observe contention across multiple senders.

4. Preserving In-Order RoCEv2 Transmission

#

Traditional RoCEv2 generally relies on ordered packet delivery within a reliable connection.

Changing the priority of packets while earlier packets remain in flight can cause packets from a later transmission segment to overtake earlier packets in the network.

This can introduce reordering and trigger expensive recovery behavior.

STORM therefore includes conservative priority-upgrade protection.

A QP cannot arbitrarily move to a higher priority while packets associated with its previous priority are still in flight.

These restrictions help preserve the ordering expectations of mainstream RoCEv2 implementations.

For newer RDMA transports that support out-of-order delivery, some of these restrictions can be relaxed or disabled.

This makes the scheduling architecture more flexible for future RDMA systems.

5. Adapting to Different Priority Resources

#

Not every data center switch provides the same number of usable priority classes.

STORM supports different configurations depending on available hardware resources.

Minimum two-priority configuration

Clusters with limited priority resources can remove the dedicated P1 class and distinguish traffic using only P2 and P3.

Extended priority configuration

When additional priority classes are available, P1 can be divided into multiple tiers based on short-message sizes.

Additional active slots can also be assigned to long-message traffic.

This flexibility allows the scheduling mechanism to adapt to the capabilities of different network fabrics.

🔧 FPGA Implementation and Hardware Overhead

#

The STORM research team implemented a hardware prototype using Xilinx Alveo U250 FPGA NICs.

The prototype builds on Coyote, an open-source FPGA-based RDMA NIC implementation supporting RoCEv2.

The implementation introduces the scheduling counters, comparisons, per-QP state, priority tagging, and receiver-side active-set management required by the design.

Its reported FPGA resource consumption is extremely small:

One important architectural decision is the separation between the packet-processing fast path and the control path.

Packet-Processing Fast Path

#

The data path performs only lightweight operations:

• Read scheduling metadata.
• Perform table lookups.
• Attach priority information.
• Execute simple eligibility and protection checks.

These operations are designed to remain constant-time, avoiding expensive scheduling computation for every packet.

Asynchronous Control Path

#

The control path handles discrete events such as:

• Work-request submission.
• Transaction completion.
• QP backlog updates.
• Active-set changes.
• Receiver-side scheduling decisions.

Separating these operations prevents asynchronous control events from introducing unnecessary processing overhead into the packet fast path.

This architectural separation is particularly important for high-speed FPGA NICs, where additional scheduling logic must not become a bottleneck for packet processing.

📊 Experimental Results: Cloud Traffic and AI Training

#

The researchers evaluated STORM using large-scale ns-3 simulations and FPGA hardware testbeds.

The comparison included several existing approaches:

• Fair sharing
• DCQCN
• PIAS
• Homa
• STORM-soft, a software-only variant without the complete receiver-side scheduling mechanism

The evaluation covered conventional cloud traffic, distributed storage-style communication, and large-scale AI training workloads.

1. Cloud Workload Performance

#

For cloud request-response workloads, STORM significantly improves flow completion performance under high network utilization.

Compared with fair sharing, the reported results show:

• Up to 90% lower average flow completion time slowdown.
• Up to 90% lower P99 flow completion time slowdown.

FCT slowdown measures how much longer a flow takes to complete compared with its ideal completion time when running without competing traffic.

Average and P99 slowdown therefore provide different perspectives on scheduling performance: average slowdown reflects overall efficiency, while P99 slowdown highlights the experience of the most delayed flows.

The results demonstrate the value of prioritizing short remaining transfers while accounting for requests that block subsequent work.

Another important finding comes from STORM-soft.

Without complete receiver-side active-set management, the software-only implementation experiences a substantial performance decline under high network load.

This indicates that sender-side priority marking alone is insufficient.

Receiver-side coordination is a critical component of the design because it helps manage contention among multiple senders targeting shared resources.

2. Distributed AI Training Performance

#

The researchers also evaluated STORM using distributed training workloads based on:

• GPT-3 13B
• GPT-3 175B
• LLaMA-7B
• LLaMA-65B

The experiments used NCCL collective communication patterns to represent communication behavior in distributed AI training.

Compared with baseline fair sharing, STORM reduced end-to-end training iteration time by up to 12%.

The improvement comes from addressing communication dependencies rather than simply increasing the throughput of individual flows.

In distributed training, a delayed collective participant can hold up the entire operation. STORM uses QP backlog to identify communication requests that may be blocking subsequent work and gives them higher scheduling priority.

This helps reduce straggler-related delays and shorten communication pipeline bubbles.

The evaluation also compared the default implementation with STORM-OOO, which disables conservative in-order transmission protection.

STORM-OOO provided only marginal additional improvements over the original implementation.

This suggests that the default design already captures most of the scheduling benefits while preserving compatibility with mainstream ordered RoCEv2 transport.

The FPGA hardware testbed further demonstrated performance improvements in collective communication completion times, with reported speedups ranging from 8% to 53% across the evaluated configurations.

3. Incremental Deployment

#

A practical network scheduling mechanism must work in environments where replacing every NIC or upgrading every host is not immediately possible.

STORM supports incremental deployment.

The evaluation indicates that enabling STORM on only approximately 25% of hosts can already produce measurable improvements in average and tail latency across the cluster.

This is important because it reduces the operational barrier to adoption.

Rather than requiring a complete infrastructure replacement, operators can introduce STORM progressively and evaluate its impact.

4. Coexistence With Existing Congestion Control

#

STORM is designed to coexist with traditional congestion-control mechanisms.

The two systems address different problems:

This distinction makes STORM complementary to rate-based congestion-control algorithms such as DCQCN.

Rather than replacing established congestion-control mechanisms, STORM introduces a request-scheduling layer that can improve workload completion efficiency.

5. Compatibility With Newer RDMA Transports

#

Although the prototype is based on RoCEv2, the underlying scheduling idea is not limited to strictly ordered transport.

Its use of NIC-visible request size and QP backlog can also fit newer RDMA architectures that tolerate out-of-order delivery and multipath transmission.

This makes the design conceptually relevant to transport developments such as Amazon SRD, Google Falcon, and Ultra Ethernet Consortium (UEC) technologies.

However, compatibility at the architectural level should not be confused with a production integration or hardware validation against every individual transport implementation.

⚠️ Limitations and Remaining Challenges

#

Despite its promising results, STORM has several limitations.

1. Potential Starvation of Low-Priority Traffic

#

STORM relies on network priority mechanisms, including strict-priority scheduling behavior.

If high-priority traffic continuously saturates the available bandwidth, low-priority traffic may experience starvation.

This is a fundamental risk of aggressive priority scheduling.

Possible mitigation mechanisms include:

• Minimum bandwidth guarantees for lower-priority queues.
• Scheduling timeouts.
• Additional fairness mechanisms.
• Limits on continuous high-priority activity.

A production deployment must balance urgency-based scheduling with long-term resource fairness.

2. Limited Application-Level Dependency Awareness

#

STORM relies on local NIC information rather than complete application dependency graphs.

Although QP backlog provides a useful indication of blocked work, it cannot always identify the globally critical transaction.

For example, storage operations involving several backend servers may have complex cross-QP dependencies.

The NIC may observe the backlog of one QP but cannot necessarily determine whether that QP belongs to the most critical application-level coflow.

The design is therefore a lightweight approximation of dependency-aware scheduling rather than a complete replacement for application-level coordination.

3. Limited Number of Network Priority Classes

#

Switch hardware generally supports only a limited number of traffic priority classes.

STORM can distinguish requests internally with more precision than the network can necessarily express.

Mapping the scheduling decision into a small number of priority levels inevitably introduces some loss of precision.

Additional hardware priority resources could improve flexibility, but they may not be available in every deployment.

4. Dependence on Accurate Logical Transaction Grouping

#

STORM’s scheduling accuracy also depends on distinguishing logical transactions from their underlying work-request fragments.

Incorrect grouping can distort backlog measurements and therefore affect scheduling decisions.

Providing reliable transaction identifiers can improve grouping accuracy, but this may require additional integration with communication libraries or frameworks.

These limitations highlight an important trade-off: STORM deliberately avoids requiring extensive application-level information, accepting a local approximation of workload dependencies in exchange for simpler deployment and low hardware overhead.

🔍 Why STORM Matters for the Future of RDMA Scheduling

#

The central contribution of STORM is not simply another packet-priority algorithm.

It changes the information used to make scheduling decisions.

Traditional size-aware scheduling asks:

Which request has the least data remaining?

STORM asks a more comprehensive question:

Which request has the most subsequent work waiting behind it, and how much data remains before that work can progress?

This distinction is particularly important for modern distributed systems.

In independent cloud workloads, prioritizing short remaining requests helps reduce completion latency.

In storage services, prioritizing transactions that block queued operations can accelerate the completion of larger application requests.

In distributed AI training, prioritizing communication associated with deeper dependency queues can help reduce straggler delays and eliminate pipeline bubbles.

What makes this approach practical is that it does not require a global scheduling controller or extensive application-level instrumentation.

The RNIC already knows the request size and has visibility into the QP’s queued work. STORM turns these existing signals into actionable scheduling decisions through a lightweight hardware architecture.

Its separation of sender-side priority assignment and receiver-side active-set management also addresses an important deployment challenge: the sender can make immediate decisions, while the receiver can coordinate long transfers using broader traffic visibility.

The reported results—up to 90% lower average and P99 FCT slowdown for cloud workloads and up to 12% shorter distributed training iterations—show that better scheduling can improve application performance without necessarily increasing raw link bandwidth.

For data centers increasingly dominated by distributed storage, large-scale GPU clusters, and communication-intensive AI workloads, this is an important direction for RDMA optimization.

The next stage of RDMA performance improvement may not come solely from transmitting more data per second. It may come from ensuring that the data everyone is waiting for arrives first.

📚 Related RDMA and Networking Resources

#

For additional technical background, the following resources provide useful context:

• RDMA Explained: How Remote Direct Memory Access Works — KAD — Covers RDMA fundamentals, RoCEv2, Queue Pairs, memory registration, and RDMA operations.
• Linux Driver mmap Explained: Zero-Copy User–Kernel I/O — PUDN — Explains zero-copy memory mapping and low-overhead data exchange between user space and kernel space.

For the original research, see STORM: Enabling Traffic Scheduling for RDMA, published in the Proceedings of ACM SIGCOMM 2026.