NVIDIA BlueField-4 CMX Targets Long-Context AI Inference
NVIDIA is introducing the BlueField-4-powered CMX context memory storage platform as a dedicated infrastructure tier for the rapidly growing demands of long-context and agentic AI inference.
As AI agents operate across longer sessions and increasingly complex workflows, inference context can expand to millions of tokens. The resulting Key-Value (KV) cache becomes a critical performance resource that must be retained, shared, and reused without repeatedly recomputing historical context.
Traditional memory and storage hierarchies are poorly suited to this workload. GPU HBM provides exceptional latency and bandwidth but remains capacity-constrained, while conventional shared storage provides scale and durability at significantly higher latency and power cost.
NVIDIA CMX addresses this gap by introducing a new G3.5 context tier between local memory and general-purpose shared storage. Built around the NVIDIA BlueField-4 data processor and connected through Spectrum-X Ethernet, the platform is designed to provide high-bandwidth, low-latency access to reusable KV cache at AI-pod scale.
NVIDIA claims the architecture can deliver up to 5x higher sustained tokens per second (TPS) and up to 5x greater power efficiency than traditional storage approaches for targeted long-context and agentic workloads.
🧠 Long-Context AI Is Reshaping the Memory Hierarchy #
AI systems are moving beyond conventional stateless chatbot interactions toward multi-turn, tool-using, and agentic workflows.
Modern foundation models can operate with context windows spanning millions of tokens, while increasingly sophisticated systems use reinforcement learning, test-time scaling, and persistent agent memory to perform more complex reasoning tasks.
For transformer-based inference, much of this retained context is represented by KV cache. Instead of recomputing previous tokens for every generation step, the system stores intermediate attention state and reuses it during subsequent inference.
As sequence lengths and session lifetimes increase, KV cache capacity grows accordingly.
KV Cache Is Both Critical and Recomputable #
KV cache presents an unusual infrastructure challenge because it is simultaneously performance-critical and ephemeral.
Unlike enterprise databases, logs, or business records, KV cache is derived data. Losing it does not necessarily result in permanent data loss because the underlying context can be reconstructed.
However, reconstructing large amounts of context can be computationally expensive. Repeatedly rematerializing KV state increases latency, consumes GPU cycles, and raises energy consumption.
This creates a new optimization target for AI infrastructure: retain reusable KV cache close enough to the compute fabric to avoid unnecessary recomputation, while avoiding the cost of storing all of it in expensive GPU HBM.
🗄️ Existing Storage Tiers Leave a Critical Gap #
Modern AI infrastructure can be viewed as a hierarchy ranging from extremely fast GPU memory to high-capacity shared storage.
NVIDIA describes four primary tiers:
- G1 — GPU HBM: Hot KV cache used directly during active token generation.
- G2 — System DRAM: Intermediate capacity for staging and buffering KV data.
- G3 — Local SSD: Warm KV cache that can be reused over shorter periods but remains tied to individual nodes.
- G4 — Shared storage: Durable storage for history, artifacts, logs, and other data that does not require immediate inference access.
Each tier represents a different compromise between latency, capacity, cost, and persistence.
GPU HBM provides the highest performance but is expensive and limited in capacity. As KV cache exceeds HBM capacity, data must move farther away from the GPU. The resulting increase in latency and data movement can reduce inference efficiency.
Pushing frequently reused KV cache all the way into conventional shared storage creates another problem. Although G4 provides massive capacity and durability, its latency and energy characteristics are poorly matched to inference context that may need to be accessed repeatedly during active workloads.
Why a Dedicated G3.5 Tier Matters #
The gap between local high-performance memory and large-scale shared storage becomes increasingly important as AI context grows.
A dedicated intermediate tier can retain large volumes of reusable KV cache while providing substantially better bandwidth and latency characteristics than conventional enterprise storage.
This is the role NVIDIA assigns to CMX.
⚡ NVIDIA CMX Introduces a Dedicated Context Memory Tier #
The NVIDIA CMX context memory storage platform uses the NVIDIA STX reference architecture and BlueField-4 data processors to establish a pod-level context memory tier.
The platform effectively introduces a G3.5 layer between local SSDs and conventional shared storage.
This Ethernet-attached flash tier is specifically optimized for KV cache rather than traditional enterprise data workloads. It provides large shared capacity that can retain context after it has been evicted from GPU HBM and system memory.
The objective is to make KV cache a shared infrastructure resource rather than a collection of isolated caches tied to individual inference nodes.
CMX is designed to provide petabytes of shared capacity per GPU pod while maintaining the bandwidth necessary to move context back toward GPU and host memory as required.
Context Prestaging Reduces Decode Stalls #
The architecture relies heavily on proactive KV cache movement.
Inference orchestration software can identify KV blocks that will be required during subsequent decode operations and prestage them from CMX into higher-speed memory tiers.
By moving context ahead of demand, the system can reduce the likelihood that GPUs will stall while waiting for historical context to arrive.
This is particularly important for long-context and agentic workloads, where repeated access to previous reasoning states can otherwise create substantial data-movement overhead.
NVIDIA says this approach can enable up to 5x higher sustained TPS in targeted long-context and agentic inference scenarios.
🔗 BlueField-4 Offloads the KV I/O Pipeline #
BlueField-4 serves as the core data-processing engine behind CMX.
The processor combines high-speed networking, multi-core NVIDIA CPU resources, high-bandwidth memory, and dedicated acceleration engines for storage and data services.
This allows CMX to handle much of the KV cache data path without consuming significant host CPU resources.
BlueField-4 can accelerate NVMe and NVMe-oF operations while supporting security and data-integrity functions such as encryption and CRC protection at line rate.
The result is a storage architecture where KV movement, protocol processing, and control-plane operations can be handled closer to the storage and network fabric rather than being serialized through the host CPU.
DOCA Memos Provides KV-Aware Data Movement #
NVIDIA’s DOCA Memos framework adds a KV-aware communication and storage layer designed specifically around the characteristics of inference context.
Rather than treating KV cache as generic block or file data, the framework recognizes KV blocks as first-class inference resources that can be shared, placed, and retrieved according to workload behavior.
DOCA Memos works with inference frameworks and BlueField-4 to move KV cache efficiently between flash storage and compute resources.
The architecture also supports open interfaces intended to allow storage partners to develop solutions around the new G3.5 context tier.
🌐 Spectrum-X Connects CMX to Rubin AI Compute #
NVIDIA Spectrum-X Ethernet provides the networking fabric connecting CMX storage resources with Rubin compute nodes.
The platform uses RDMA-oriented networking to provide high-bandwidth and predictable access to shared KV cache.
Spectrum-X combines congestion control, adaptive routing, and optimized RoCE to reduce packet loss, jitter, and tail latency under demanding AI traffic conditions.
This consistency is particularly important for distributed inference. A context tier is only useful if multiple compute nodes can access shared KV data without introducing unpredictable delays that negate the performance benefits of reuse.
By integrating CMX into the same AI-optimized network fabric used by the compute infrastructure, NVIDIA aims to make context memory a scalable resource across the entire AI pod.
🔥 CMX Targets Higher Performance per Watt #
Power efficiency is becoming a major constraint for large-scale AI infrastructure.
Traditional storage systems often include substantial overhead for durability, metadata management, replication, consistency mechanisms, and other enterprise features. Those capabilities are valuable for persistent business data but can be excessive for derived and reconstructable KV cache.
CMX takes a different approach by treating KV cache as an AI-native data class with different requirements.
Because KV data can be regenerated, the architecture can prioritize throughput, latency, power efficiency, and capacity over the extensive durability mechanisms typically associated with enterprise storage.
NVIDIA claims CMX can provide up to 5x better power efficiency than general-purpose storage approaches for the targeted KV cache workload.
The benefits can extend beyond the storage system itself. Faster context retrieval and reliable prestaging can prevent GPUs from sitting idle while waiting for historical context, while avoiding unnecessary recomputation reduces wasted accelerator cycles.
The result is a focus on useful tokens generated per unit of infrastructure power rather than storage performance in isolation.
🏭 CMX Extends NVIDIA Vera Rubin AI Factories #
The CMX platform is designed as part of NVIDIA’s broader Vera Rubin AI infrastructure architecture.
Vera Rubin organizes AI infrastructure into compute, networking, and storage components that can be assembled into scalable AI factories supporting pretraining, post-training, test-time scaling, and real-time agentic inference.
Within this architecture, CMX provides a dedicated context layer for large-scale inference.
The additional tier allows AI infrastructure operators to separate workloads according to their actual data requirements:
- GPU HBM can remain focused on active, latency-critical KV cache.
- System memory can handle staging and near-term context.
- Local storage can retain node-local warm data.
- CMX can provide shared, high-bandwidth context memory across the pod.
- Conventional shared storage can remain focused on durable artifacts and long-term records.
This separation can reduce pressure on both GPU memory and traditional storage infrastructure.
📊 Improving Agentic AI Through Shared KV Reuse #
The architectural significance of CMX extends beyond storage capacity.
Agentic AI systems increasingly operate as persistent workflows rather than isolated inference requests. Agents may repeatedly access previous conversations, tool outputs, intermediate reasoning states, and other context across multiple inference steps.
When that context can be reused efficiently, the system avoids repeatedly reconstructing the same state.
CMX turns this reusable context into a shared resource that can move between inference services and nodes within the AI pod.
NVIDIA Dynamo and the NVIDIA Inference Transfer Library (NIXL) can coordinate context movement across the hierarchy, while topology-aware orchestration can account for KV locality when assigning workloads.
This creates a continuum of storage and memory tiers where context can be placed according to its latency requirements and likelihood of reuse.
💡 A New Storage Model for AI-Native Inference #
NVIDIA CMX represents a broader shift in AI infrastructure design: treating inference context as a specialized data category rather than forcing it into conventional memory and storage architectures.
The fundamental challenge is no longer simply storing more data. AI systems need to retain increasingly large amounts of reusable context while keeping that context close enough to the compute fabric to maintain high inference throughput.
A dedicated G3.5 tier addresses this problem by combining flash capacity, high-bandwidth networking, BlueField-4 data processing, and KV-aware orchestration.
If the reported performance and efficiency targets translate into production workloads, CMX could become an important architectural component for scaling long-context and agentic inference.
The underlying principle is straightforward: keep the hottest context in GPU memory, move reusable context into a purpose-built shared tier, and reserve conventional storage for data that genuinely requires long-term persistence.
For AI factories operating at gigascale, that distinction could determine how efficiently infrastructure converts power and compute capacity into useful tokens.