NVIDIA Vera Rubin Benchmark: 30x AI Throughput per Megawatt
NVIDIA’s next-generation Vera Rubin NVL72 platform is generating significant attention with a benchmark claim that targets one of the most important metrics in modern AI infrastructure: throughput per megawatt.
In August, NVIDIA published benchmark results for Vera Rubin using the DeepSeek-V4-Pro model and an AgentX workload. According to the company’s data, Vera Rubin NVL72 achieved up to 30x higher throughput per megawatt than the previous-generation GB300 NVL72.
The comparison is particularly notable because GB300 is not an obsolete platform. It represents NVIDIA’s current flagship-generation architecture preceding Vera Rubin.
A generational improvement of this magnitude would have implications far beyond raw accelerator performance. In large AI data centers, power availability is increasingly one of the primary constraints on deploying additional compute capacity.
If the benchmark results translate into production workloads, improving the amount of AI inference that can be delivered from each megawatt could materially change the economics of AI infrastructure.
However, the headline number needs to be interpreted carefully. It comes from a specific workload and configuration, and the results are vendor-reported pre-production figures.
⚡ What Does Throughput per Megawatt Actually Mean? #
Traditional accelerator benchmarks often emphasize metrics such as FLOPS, tokens per second, or latency.
For large-scale AI inference, however, another metric is becoming increasingly important:
How much useful AI work can the data center produce for a fixed amount of electrical power?
This is the concept behind throughput per megawatt.
An AI factory containing tens of thousands of accelerators can consume enormous amounts of electricity. At sufficiently large scale, a cluster can require power comparable to that of a small city.
Consequently, the relevant economic question is not simply:
How fast is one GPU?
It is:
How many tokens can the entire infrastructure generate for every megawatt of available power?
That metric directly influences several operational variables.
Cost per generated token #
Higher throughput per megawatt means more inference work can be completed for the same electricity budget.
If all other variables remain constant, this can reduce the energy component of the cost per token.
Infrastructure utilization #
A fixed power allocation can support more inference capacity when the underlying hardware is more energy efficient.
This becomes especially important in regions where obtaining additional grid capacity is more difficult than purchasing additional accelerators.
Data center economics #
AI infrastructure operators must consider power, cooling, networking, real estate, and accelerator costs simultaneously.
Improving compute output without proportionally increasing power consumption can increase the amount of useful AI work generated from existing data-center infrastructure.
📊 NVIDIA’s Reported Efficiency Comparison #
According to the benchmark figures described by NVIDIA, the platforms show the following relative progression:
| Platform | Reported Throughput per Megawatt | Relative Position |
|---|---|---|
| H200 NVL8 | Baseline | Previous generation |
| GB300 NVL72 | Up to 80x H200 | Current flagship |
| Vera Rubin NVL72 | Up to 30x GB300 | Next generation |
If these figures are directly multiplicative under the stated benchmark conditions, the implied Vera Rubin result would be approximately 2,400x the H200 baseline:
80 × 30 = 2,400
That arithmetic illustrates why the headline is so striking.
However, the multiplication should not be interpreted as a universal claim that Vera Rubin is 2,400x faster or more efficient than H200 across all AI workloads.
The figures represent specific benchmark configurations and workload conditions.
The distinction between throughput per megawatt and raw compute performance is particularly important.
A 30x increase in tokens-per-megawatt does not mean that an individual Vera Rubin GPU performs 30x more computation than a GB300 GPU.
It indicates that the complete tested system can produce substantially more benchmark throughput for a given power envelope.
🧠 Why Use DeepSeek-V4-Pro for the Benchmark? #
The choice of workload is an important part of interpreting the result.
The benchmark uses DeepSeek-V4-Pro, described in the source material as a 1.6-trillion-parameter Mixture-of-Experts (MoE) model.
MoE architectures can be particularly demanding from a systems perspective because the total parameter count can be much larger than the number of parameters activated for an individual inference operation.
MoE changes the hardware bottleneck #
A simplified MoE inference path can be represented as:
Input tokens → Router → Expert selection → Distributed computation → Expert aggregation → Output
Only a subset of experts is activated for each token, but the system still needs to manage a very large overall parameter set.
This places substantial demands on:
- GPU memory capacity
- Memory bandwidth
- Inter-GPU communication
- Collective operations
- Parameter placement
- Scheduling
- Network topology
- Inference software
Consequently, large-scale MoE inference is not simply a matter of maximizing arithmetic throughput.
The accelerator cluster must efficiently move data and coordinate computation across many GPUs.
NVLink becomes strategically important #
NVIDIA’s approach to large MoE models depends heavily on high-bandwidth accelerator interconnects.
The Vera Rubin platform introduces NVLink 7.0, which is designed to provide substantially greater communication bandwidth between GPUs than previous generations.
Higher interconnect bandwidth can reduce communication bottlenecks when model execution is distributed across many accelerators.
This is especially relevant when multiple GPUs must cooperate on the same inference workload.
🏗️ Vera Rubin’s Full-Stack Approach #
A 30x system-level efficiency improvement cannot reasonably be attributed to a single component.
The reported gain is the result of optimization across several layers of the infrastructure stack.
GPU compute #
Compared with the Blackwell generation, Vera Rubin is designed to provide significantly higher computational throughput.
The source material describes approximately 2x single-GPU FP8 compute performance relative to the previous generation.
Higher compute density allows more inference work to be completed within the same physical infrastructure footprint.
HBM4 memory #
Vera Rubin introduces HBM4, providing substantial improvements in memory capacity and bandwidth.
For large AI models, memory is often as important as arithmetic throughput.
Insufficient memory bandwidth can leave compute units underutilized, while insufficient capacity can force model parameters to be distributed across additional accelerators.
Higher-bandwidth HBM therefore contributes directly to overall accelerator utilization.
NVLink 7.0 #
The platform also introduces NVLink 7.0 as a higher-bandwidth GPU interconnect.
For multi-GPU workloads, especially large MoE models, communication between accelerators can become a significant portion of total execution time.
Increasing interconnect bandwidth helps reduce this bottleneck.
🖥️ NVL72: Turning 72 GPUs Into a Unified Computing System #
The Vera Rubin NVL72 architecture connects 72 GPUs through a high-bandwidth NVLink fabric.
The significance of NVL72 is not simply that it contains 72 individual accelerators.
The architecture is designed to make the GPUs operate as a tightly coupled computing domain.
Large shared model capacity #
Distributing model parameters across many GPUs allows the system to execute models that would not fit within the memory capacity of a single accelerator.
This becomes increasingly important as model sizes continue to grow.
High-bandwidth communication #
A large distributed model requires frequent communication between accelerators.
A high-bandwidth interconnect reduces the relative cost of these communication operations and can improve overall accelerator utilization.
System-level efficiency #
NVIDIA’s architecture targets substantially higher efficiency than conventional small multi-GPU servers.
The key idea is that the performance of a large AI system depends on more than the capabilities of individual GPUs.
The system must be optimized as a single computing platform:
GPU compute + HBM + interconnect + networking + software + cooling + power delivery
💻 Software Is a Critical Part of the Performance Equation #
Hardware improvements alone cannot explain large gains in AI inference efficiency.
The software stack determines how effectively the hardware is utilized.
NVIDIA’s AI software ecosystem includes components spanning:
- CUDA
- GPU drivers
- Communication libraries
- Tensor and inference libraries
- Model optimization tools
- Runtime systems
- Networking software
Large-scale inference requires these components to be optimized together with the underlying accelerator architecture.
For an MoE workload, software must efficiently manage expert routing, parameter placement, communication, scheduling, memory movement, and execution.
A theoretically powerful GPU can still deliver poor real-world utilization if the software stack cannot keep the hardware occupied.
This is why NVIDIA’s long-established CUDA ecosystem remains strategically important when evaluating new accelerator generations.
🌡️ Power and Cooling Are Part of the Compute Architecture #
The efficiency equation does not end at the GPU.
A large AI system consumes power across the entire infrastructure:
Grid → Power conversion → Rack → Accelerator → Computation
Every conversion stage introduces losses.
Cooling systems also consume electricity, which means thermal design directly affects the amount of power available for useful computation.
Power conversion efficiency #
Improving power-delivery efficiency reduces the amount of electrical energy lost before it reaches the compute hardware.
Even relatively small percentage improvements can become significant when applied across megawatts of infrastructure.
Liquid cooling #
As accelerator thermal density increases, conventional air cooling becomes increasingly difficult to scale efficiently.
Liquid cooling can remove heat more effectively and can reduce the electrical overhead associated with moving large quantities of air.
The resulting power savings can effectively return additional energy to the compute budget.
Data-center-level optimization #
The ultimate target is therefore not merely GPU efficiency.
It is grid-to-token efficiency.
That requires optimizing the complete chain from electrical power entering the data center to useful AI output leaving the system.
📈 Potential Impact on AI Infrastructure #
If NVIDIA’s reported benchmark results are reproduced under independent testing and across representative production workloads, the implications could be substantial.
Lower inference energy costs #
A large increase in throughput per megawatt means more tokens can be generated from the same power budget.
The energy cost associated with each generated token could therefore decrease significantly.
However, electricity is only one component of inference cost. Hardware depreciation, networking, cooling, data-center operations, software, and capital costs also contribute to the final cost per token.
More output from existing power capacity #
Power availability has become a major constraint for new AI data centers.
A facility that already has a fixed power allocation cannot necessarily expand simply by purchasing more GPUs.
Higher energy efficiency changes that equation.
If the same power envelope can produce substantially more inference throughput, operators can increase useful AI capacity without increasing grid consumption proportionally.
Pressure on competing accelerators #
Large improvements in system-level efficiency would also increase competitive pressure on other accelerator platforms.
This includes:
- AMD Instinct
- Intel Gaudi
- Google TPU
- AWS Trainium
- Microsoft Maia
- Custom ASICs developed by major AI companies
Competitors are increasingly optimizing not just raw compute, but complete cost-per-token and performance-per-watt metrics.
⚠️ Why the 30x Figure Needs Context #
The headline number is impressive, but benchmark interpretation requires discipline.
It represents a specific workload #
The reported 30x improvement comes from a particular benchmark configuration involving:
- AgentX workloads
- DeepSeek-V4-Pro
- Specific model settings
- Specific batch sizes
- NVL72 system configurations
Results can vary substantially with workload characteristics.
Dense models may behave differently from MoE models, while training workloads can have very different bottlenecks from inference.
Vera Rubin is pre-production #
The reported results are based on pre-production hardware.
Production silicon can differ from early samples in clock behavior, thermal characteristics, firmware, software maturity, and system-level optimization.
Real-world performance therefore needs to be measured on production deployments.
Vendor benchmarks require independent validation #
NVIDIA’s own benchmark results represent optimized configurations selected by the vendor.
That does not make the measurements irrelevant, but it does mean they should be evaluated alongside independent testing.
The source material also cites independent CoreWeave testing of DeepSeek-R1 on Vera Rubin, which reportedly demonstrated approximately a 10x throughput-per-megawatt improvement over Grace Blackwell.
The difference between a vendor-reported 30x result and an independent 10x result illustrates why workload definitions, configurations, and measurement methodology matter.
The important question is not simply whether Vera Rubin can achieve 30x under one benchmark.
It is whether similar gains remain visible across the diverse workloads, model architectures, utilization levels, and operating conditions encountered in production AI factories.
🔭 From GPU Performance to AI Factory Efficiency #
The most important aspect of Vera Rubin’s reported performance is the shift in how AI accelerators are evaluated.
The industry is gradually moving beyond:
How many FLOPS can this GPU deliver?
toward:
How many useful tokens can this infrastructure produce per megawatt, per rack, and per dollar of capital expenditure?
That shift reflects the economics of modern AI.
At hyperscale, electricity, cooling, networking, and physical infrastructure can become just as important as the accelerator itself.
Vera Rubin’s architecture therefore represents more than another GPU-generation upgrade. It reflects NVIDIA’s broader strategy of optimizing the entire AI factory:
GPU compute → HBM4 → NVLink 7.0 → NVL72 → CUDA → networking → cooling → power infrastructure
If the reported results translate into sustained production performance, the consequences could extend across the entire AI infrastructure market.
The 30x figure should not be interpreted as a universal performance multiplier. It is a benchmark result tied to a specific workload and system configuration.
Nevertheless, even a substantially smaller real-world improvement would be significant at data-center scale.
The long-term competitive metric for AI infrastructure may ultimately be neither raw GPU performance nor peak FLOPS, but useful AI output generated for every unit of power and capital invested.