NVIDIA Rubin GPU Architecture: A Deep Technical Breakdown

NVIDIA Rubin GPU Architecture: A Deep Technical Breakdown

NVIDIA has published detailed technical information about its next-generation Rubin GPU architecture, positioning it as the core compute engine of the Vera Rubin AI platform and a fundamental redesign for the emerging Agentic AI workload.

Unlike architectures primarily optimized around conventional LLM training and single-request inference, Rubin is designed around the characteristics of autonomous AI agents: multi-step reasoning, tool invocation, long-context processing, dynamic expert routing, and continuous high-concurrency inference.

The resulting architecture is therefore not simply a generational increase in GPU compute. NVIDIA has redesigned memory, interconnects, kernel scheduling, compression, power management, and rack-scale infrastructure around the goal of maximizing useful token throughput while minimizing data movement and idle time.

NVIDIA claims that the Vera Rubin platform can deliver a 10x generational improvement in Agentic AI inference performance per unit of energy on its internal 2-trillion-parameter MoE workloads compared with Blackwell.

🤖 Agentic AI Changes the Compute Problem

#

Traditional LLM inference generally follows a relatively straightforward request-response pattern. Agentic AI introduces a much more complex execution model in which an AI system can repeatedly reason, call external tools, inspect results, revise its plan, and continue execution.

This changes the primary performance bottlenecks.

Multi-step continuous reasoning

#

An agent may execute dozens or even hundreds of sequential inference operations during a single task. Each step can introduce additional kernel launches, synchronization points, memory operations, and scheduling dependencies.

Consequently, maximizing peak FLOPS alone does not guarantee high end-to-end throughput. The accelerator must also minimize the idle periods created between dependent operations.

Million-token context windows

#

Coding agents, enterprise knowledge systems, and complex multi-turn workflows can require context windows ranging from hundreds of thousands of tokens to more than one million tokens.

At these scales, KV cache capacity and memory bandwidth become major constraints. Keeping frequently accessed KV state close to the GPU can avoid expensive transfers to slower host memory.

Trillion-parameter MoE models

#

Mixture-of-Experts architectures allow models to scale to enormous parameter counts while activating only a subset of experts for each token.

However, dynamic expert routing creates substantial communication requirements. Tokens and activations must move between GPUs, while distributed systems must synchronize the results.

At sufficient scale, inter-GPU communication can become a larger bottleneck than raw tensor compute.

Continuous high-concurrency inference

#

Agentic AI infrastructure is expected to operate continuously, with large numbers of concurrent sessions running across AI factories and cloud clusters.

This shifts the economic metric from peak compute to useful token production per unit of power.

For hyperscalers, tokens per watt, memory efficiency, and rack-level utilization can therefore matter as much as theoretical FLOPS.

🧩 Rubin GPU Architecture and Hardware Specifications

#

Rubin continues NVIDIA’s chiplet-oriented packaging strategy. The GPU combines two reticle-limited compute dies through NVIDIA’s High-Bandwidth Interface, or NV-HBI, into a single package.

The architecture reportedly uses TSMC’s 3nm process and substantially expands transistor count, compute resources, memory capacity, and interconnect bandwidth compared with Blackwell.

Rubin versus Blackwell

#

Metric	Blackwell GB300	Rubin GPU	Improvement
Transistors	208 billion	336 billion	+62%
Streaming Multiprocessors	192	224	+17%
Tensor Cores	768	896	+17%
NVFP4 Peak Inference	10 PFLOPS	50 PFLOPS	5x
HBM Capacity	192GB HBM3e	288GB HBM4	+50%
Memory Bandwidth	8TB/s	22TB/s	2.8x
GPU Interconnect	1.8TB/s NVLink5	3.6TB/s NVLink6	2x
CPU-GPU Interconnect	—	1.8TB/s NVLink-C2C	New
PCIe	Gen5 x16	Gen6 x16	Up to 256GB/s
Softmax FP16/BF16 Throughput	1x	4x	4x

The most important change is that NVIDIA is increasing performance across several dimensions simultaneously. Rubin combines substantially more transistor resources with faster memory, greater memory capacity, higher inter-GPU bandwidth, and specialized execution improvements for attention and MoE workloads.

NV-HBI chiplet architecture

#

Two compute dies are connected through NV-HBI to form a single logical GPU package.

This approach allows NVIDIA to bypass the physical reticle-size limitations associated with building an increasingly large monolithic die.

The architecture is organized around Graphics Processing Clusters and a large unified L2 cache, with the GigaThread engine responsible for distributing work across the GPU.

Hardware-level Multi-Instance GPU virtualization also provides task isolation for multi-tenant environments.

Third-generation Transformer Engine

#

The third-generation Transformer Engine is one of Rubin’s primary performance components.

It supports multiple numerical formats, including:

• NVFP4
• FP8
• BF16
• FP16
• TF32
• FP32

The architecture can dynamically use lower precision where appropriate to reduce memory traffic and power consumption while retaining higher precision where model training or numerical stability requires it.

Rubin also introduces adaptive compression for activations and KV caches, reducing the amount of memory required during inference.

Next-generation Tensor Cores

#

Rubin increases matrix instruction throughput per clock and reduces the number of K-loop iterations required for matrix operations.

The reported reduction from four K-loops on Blackwell to two on Rubin lowers instruction scheduling overhead and can improve Tensor Core utilization.

This is particularly relevant to tensor-parallel distributed models, where compute efficiency can be limited by synchronization and scheduling overhead during both Prefill and Decode.

⚙️ Five Architectural Innovations for Agentic Workloads

#

NVIDIA’s Rubin design targets specific bottlenecks created by Agentic AI rather than relying exclusively on increased peak compute.

1. 🚀 Enhanced TMA and Inline Descriptor Updates

#

The Tensor Memory Accelerator plays an important role in moving multidimensional tensor data between memory and compute units.

MoE models create additional complexity because expert weights must be accessed according to dynamically changing routing decisions.

Rubin introduces inline descriptor updates that allow memory addresses and stride parameters to be modified directly during TMA execution.

This reduces the need to repeatedly update tensor metadata in memory.

Tensors sharing common layouts can also reuse descriptors, reducing metadata traffic and improving the efficiency of repeated expert-weight accesses.

The architecture is consequently better suited to dynamically loading only the expert weights required for a particular workload instead of keeping every expert resident in active memory.

2. 🧠 Activation Sparsity and Adaptive Compression

#

Long-context attention creates significant memory and compute pressure, particularly during the calculation and normalization of attention matrices.

Rubin addresses this through activation sparsity and hardware-accelerated Softmax processing.

The architecture supports 2:4 structured activation sparsity, allowing selected intermediate values to be represented using sparse data structures.

By reducing the amount of information that needs to move through the memory hierarchy, the system can lower data movement and potentially improve attention efficiency.

Rubin also accelerates exponential calculations used by Softmax, with reported improvements of up to:

• 2x FP32 throughput
• 4x BF16/FP16 throughput

These improvements target one of the recurring bottlenecks in large attention operations.

3. 🔄 Fine-Grained Dependent Kernel Triggering

#

Agentic workloads frequently contain producer-consumer dependencies.

A downstream operation may need only a portion of an upstream result before it can begin, but traditional scheduling approaches can leave compute resources idle while waiting for the complete producer operation.

Rubin introduces tile-level dependent kernel triggering.

Instead of waiting for an entire operation to finish, consumer kernels can begin processing as soon as the required local tile becomes available.

This allows computation to overlap more aggressively and keeps the GPU pipeline occupied during long sequences of dependent operations.

NVIDIA reports that the approach can increase concurrent throughput by more than 30% in targeted workloads.

4. 💾 288GB HBM4 Memory Subsystem

#

Memory bandwidth is especially important during the Decode phase of LLM inference, where workloads can become heavily memory-bound.

Rubin addresses this with 288GB of 12-Hi HBM4 per GPU.

Compared with Blackwell’s 192GB HBM3e configuration, Rubin increases local accelerator memory capacity by 50%.

Peak memory bandwidth rises to approximately 22TB/s, representing a substantial increase over Blackwell’s 8TB/s.

The larger memory pool can retain significantly more KV-cache state locally, reducing the need to offload long-context data to slower memory tiers.

For million-token Agentic workloads, this additional capacity can be particularly valuable because KV-cache requirements grow rapidly with context length and concurrency.

5. 🔗 NVLink6 and Counted Writes

#

Rubin introduces sixth-generation NVLink with up to 3.6TB/s of bidirectional GPU interconnect bandwidth.

That is approximately twice the bandwidth of the previous generation.

The platform also introduces NVLink-C2C connectivity between the Rubin GPU and Vera CPU, providing up to 1.8TB/s of bandwidth.

This high-speed CPU-GPU path is designed to support low-latency coordination for tasks such as agent scheduling, tool execution, and workload orchestration.

Counted Writes

#

Rubin also introduces hardware-level Counted Writes synchronization.

Rather than requiring CPU intervention to determine whether distributed data transfers have completed, the communication hardware tracks transmission progress directly.

This allows synchronization to occur closer to the GPU execution path and can reduce overhead during operations such as distributed All-Reduce and MoE expert routing.

NVIDIA reports up to a 60% reduction in synchronization latency for targeted distributed workloads.

🏢 Vera Rubin NVL72 Brings Rubin to Rack Scale

#

The Rubin GPU is only one component of the larger Vera Rubin platform.

A fully configured NVL72 rack integrates:

• 72 Rubin GPUs
• 36 Vera CPUs
• NVLink6 networking
• BlueField-4 DPUs
• ConnectX-9 networking
• NVLink switch infrastructure
• Liquid cooling
• Rack-level power management

The design treats the rack as a single AI computing system rather than a collection of independent servers.

DSX MaxLPS power management

#

Agentic workloads can produce highly variable power consumption because inference activity changes dynamically with reasoning depth, token generation, and concurrent agent execution.

NVIDIA’s DSX MaxLPS technology is designed to smooth these transient power fluctuations.

The reported targets include:

• 20% reduction in peak power
• 10% reduction in average power
• Up to 40% more GPUs under equivalent facility power constraints

This approach allows data centers to optimize infrastructure around actual workload behavior rather than simply designing for the theoretical maximum instantaneous load.

Warm-water liquid cooling

#

Rubin GPUs are designed for high-density liquid-cooled deployments.

The Vera Rubin rack uses approximately 45°C warm-water cooling and supports GPUs with reported TDP levels between 1,200W and 1,500W.

The MGX Gen 3 rack architecture also incorporates a cableless backplane design intended to simplify serviceability.

Hot-swappable NVLink switch trays can reportedly be replaced without interrupting ongoing cluster inference.

Confidential computing

#

The platform incorporates hardware-based Trusted Execution Environment capabilities through TEE-I/O.

The objective is to provide protection for sensitive AI workloads across storage, data transfer, and compute operations.

This is particularly important for enterprise and regulated deployments where proprietary data and model execution must remain isolated throughout the inference pipeline.

Multi-chip workload orchestration

#

The Vera CPU is responsible for tasks that extend beyond raw accelerator computation, including:

• Tool invocation
• Agent creation
• Task branching
• Scheduling
• System coordination

Additional dedicated hardware handles networking and other infrastructure operations.

This heterogeneous design allows the platform to divide AI workloads across specialized processors instead of forcing GPUs to perform every stage of the execution pipeline.

📊 Rubin’s Claimed Agentic AI Performance Advantage

#

NVIDIA’s central performance claim for Vera Rubin is based on an internal 2-trillion-parameter MoE Agentic AI benchmark.

The company reports a 10x increase in agent throughput per unit of energy compared with Blackwell.

The improvement comes from the combined effect of several architectural changes rather than a single increase in FLOPS.

Lower token production costs

#

Higher HBM capacity, greater memory bandwidth, faster GPU interconnects, and reduced synchronization overhead can lower the amount of energy required to generate each token.

For cloud operators, this directly affects the cost of running large-scale inference services.

If the claimed efficiency improvements translate to production workloads, energy consumption and hardware depreciation could become substantially lower on a per-token basis.

Trillion-parameter models in a single rack

#

The 72-GPU NVL72 configuration provides a tightly coupled memory and interconnect domain.

Each Rubin GPU contributes 288GB of HBM4 and 22TB/s of local memory bandwidth, while the rack provides high-bandwidth GPU-to-GPU communication.

This architecture is designed to keep very large MoE models within a single rack where possible, potentially reducing the amount of cross-rack networking required.

Reducing cross-rack traffic can lower network infrastructure requirements, latency, and operational complexity.

High-concurrency million-token inference

#

The combination of large HBM4 capacity, accelerated attention operations, sparse activation processing, and high-bandwidth interconnects is designed for workloads involving many simultaneous long-context sessions.

Potential applications include:

• Coding agents
• Enterprise knowledge systems
• Digital workers
• Autonomous research agents
• Large-scale tool-using AI systems

The key architectural objective is to keep compute resources continuously occupied even when individual agent tasks involve irregular sequences of dependent operations.

🎯 Rubin Represents NVIDIA’s Shift Toward Agentic AI Infrastructure

#

Hopper and Blackwell established NVIDIA’s dominance in large-scale AI training and conventional inference. Rubin extends that strategy into a workload environment where the primary challenge is no longer simply maximizing matrix throughput.

Agentic AI introduces new bottlenecks across memory capacity, memory bandwidth, synchronization, kernel scheduling, interconnects, power delivery, and rack-level orchestration.

Rubin addresses these constraints through a coordinated set of technologies:

• NV-HBI chiplet packaging
• Third-generation Transformer Engines
• Larger HBM4 memory
• Faster NVLink6 interconnects
• NVLink-C2C CPU-GPU connectivity
• Sparse activation processing
• Accelerated Softmax
• Fine-grained kernel scheduling
• Hardware synchronization
• Rack-level power management
• Liquid cooling
• Full-stack confidential computing

The broader significance of Rubin is therefore its full-system approach. NVIDIA is no longer optimizing only the GPU; it is designing the accelerator, CPU, networking, memory, cooling, power infrastructure, and software as one platform for continuous Agentic AI execution.

If NVIDIA’s internal performance claims translate to real-world workloads, Vera Rubin could establish a new benchmark for AI infrastructure economics where tokens per watt, context capacity, interconnect efficiency, and sustained agent throughput become as important as peak accelerator FLOPS.