MLPerf Inference v6.1: AMD, Blackwell and Vera Rubin Tested
MLCommons has released the results of MLPerf Inference v6.1, introducing a record-scale 512-accelerator submission, new benchmarks for retrieval-augmented generation and edge agentic inference, and the first peer-reviewed performance results for NVIDIA’s Vera Rubin platform.
The September 16, 2026 release included submissions from 30 organizations covering 120 systems and 486 data-center and edge results. The benchmark round spans everything from individual edge devices and workstation GPUs to rack-scale systems with hundreds of accelerators.
The results also highlight a changing AI inference landscape. AMD demonstrated large-scale MI355X scaling at 512 GPUs, NVIDIA’s Blackwell platform remained highly competitive across conventional deployment sizes, and Vera Rubin made its first appearance in the benchmark. Intel expanded its participation with Arc Pro B70 and additional Xeon 6 configurations.
Rather than showing a single platform dominating every scenario, MLPerf Inference v6.1 illustrates how system scale, workload characteristics, software optimization, and accelerator architecture increasingly interact to determine inference performance.
🚀 AMD MI355X Reaches a Record 512-GPU Scale #
The most visible result in MLPerf Inference v6.1 is AMD’s 512-GPU Instinct MI355X submission.
MLCommons identified it as the largest system ever submitted to MLPerf Inference, surpassing the previous record of 288 accelerators from the preceding round. The 512-GPU system was submitted by Crusoe and consisted of 64 nodes using AMD Instinct MI355X accelerators.
On the GPT-OSS-120B benchmark, the system achieved approximately:
- 5.75 million tokens/second in Offline mode
- 5.39 million tokens/second in Server mode
For DeepSeek-R1, it reached approximately:
- 2.90 million tokens/second in Offline mode
- 2.40 million tokens/second in Server mode
These results represent aggregate cluster throughput rather than the performance of an individual GPU.
The significance is therefore primarily about scale-out inference. Modern large-model serving increasingly requires multiple accelerators and nodes, making interconnects, networking, distributed scheduling, memory capacity, and software orchestration important components of total system performance.
Scaling efficiency becomes a key metric #
The benchmark also provides a useful view of how inference throughput scales as additional accelerators are added.
AMD reported 95% scaling efficiency for GPT-OSS-120B at a 72-GPU scale on the MI355X platform. NVIDIA’s 288-GPU GB300 submission, meanwhile, achieved 99% scaling efficiency, with throughput scaling close to linearly relative to a single GB300 NVL72 system.
These figures illustrate why raw accelerator throughput alone is insufficient for hyperscale inference.
As cluster size increases, communication overhead, synchronization, data movement, scheduling, and networking can reduce the theoretical benefit of adding more GPUs. High scaling efficiency indicates that the software and infrastructure stack is converting additional accelerator capacity into useful inference throughput effectively.
🖥️ NVIDIA Blackwell Remains Strong at Conventional Deployment Sizes #
While AMD’s 512-GPU configuration captured attention at extreme scale, NVIDIA’s Blackwell platform remained highly competitive in more conventional 8-GPU and 72-GPU configurations.
In the Llama 2 70B benchmark, NVIDIA’s GB300 configurations led competing systems at both 72-GPU and 8-GPU scales.
The Llama 3.1 8B results also showed a performance advantage for the 8-GPU GB300 configuration compared with an equivalently sized MI355X setup, with NVIDIA reporting an advantage of up to 17% in that configuration.
The GPT-OSS-120B results demonstrate why benchmark conclusions need to be tied to a specific deployment scale. AMD’s 512-GPU MI355X system achieved very high aggregate throughput at extreme scale, while GB300 remained highly competitive in 72-GPU and 8-GPU configurations.
MLPerf’s results therefore expose different dimensions of accelerator performance rather than reducing the comparison to a single ranking.
🔮 Vera Rubin Makes Its MLPerf Debut #
MLPerf Inference v6.1 also marks the first peer-reviewed benchmark appearance of NVIDIA Vera Rubin.
The Vera Rubin NVL72 platform participated in the DeepSeek-R1 benchmark, with submissions covering 72-GPU and 36-GPU configurations. MLCommons identified Vera Rubin as one of the new preview-category platforms appearing in this release.
According to NVIDIA’s submission highlights, the Vera Rubin NVL72 delivered up to 2.5x the DeepSeek-R1 throughput of the prior generation and up to 3.7x the throughput on Qwen3-VL.
The results are significant because they provide an early, independently benchmarked reference point for the next-generation architecture.
However, Vera Rubin remains a preview platform in this MLPerf round. As with any early benchmark submission, its results should be interpreted within the tested workloads, configurations, software versions, and benchmark rules rather than generalized to every inference workload.
🧠 New Benchmarks Reflect the Shift Toward Agentic AI #
MLPerf Inference v6.1 expands beyond traditional single-model inference with two new benchmark areas:
- End-to-End RAG for data-center deployments
- Agentic Edge Inference for single-user edge devices
The RAG benchmark evaluates a multi-stage retrieval-augmented generation pipeline rather than measuring only the final LLM. The workflow includes embedding, retrieval, re-ranking, and LLM-based answer generation.
This is closer to how enterprise RAG systems are actually constructed, where inference performance depends on interactions among multiple models and infrastructure components. MLCommons also separates document ingestion from query answering to provide more detailed performance information.
The new Edge Agentic Inference benchmark similarly reflects the move from single-turn model execution toward multi-step workloads such as agentic coding.
MLPerf has therefore started to shift from asking “How fast can this accelerator run a model?” toward more system-oriented questions such as “How efficiently can this platform execute an end-to-end AI workload?”
🖥️ AMD and NVIDIA Are Not the Only Platforms Expanding #
The v6.1 results also broaden the range of tested hardware.
AMD submitted the Instinct MI350P for the first time, while NVIDIA’s RTX PRO professional GPUs appeared in multiple workstation and server configurations. MLCommons also recorded first peer-reviewed results for AMD’s Ryzen AI Max+ 395 and Intel’s Arc Pro B70.
Intel’s Arc Pro B70 participated in workloads including:
- Llama 3.1 8B
- Llama 2 70B
- GPT-OSS-120B
- Whisper
- End-to-End RAG
The Arc Pro B70 submission used a four-GPU configuration with 128 GB of total VRAM.
Intel reported that, on the same four-GPU hardware used in MLPerf Inference v6.0, GPT-OSS-120B performance increased by 36% in Server mode and 27% in Offline mode, attributing the improvement primarily to software-stack maturation.
The broader result is important because inference hardware is increasingly extending beyond traditional data-center accelerators into workstation and edge-class products.
⚙️ Intel Xeon 6 Shows the Impact of Software Optimization #
Intel also expanded its Xeon 6 participation.
The number of submitted Xeon SKUs increased from two in v6.0 to five, while CPU inference results increased from 24 to 35. MLCommons noted that Xeon was the only standalone server CPU platform represented in this round.
Intel reported substantial gains on the Xeon 6980P using identical hardware and socket counts. In the Llama 3.1 8B Server scenario, throughput increased by 2.4x compared with its v6.0 result, while Offline throughput increased by 56%.
These improvements demonstrate why benchmark generations cannot always be interpreted as direct hardware-generation comparisons.
A substantial portion of the gain can come from improvements in kernels, compilers, runtimes, quantization, scheduling, model implementations, and other components of the software stack.
🧩 Software Optimization Is Becoming a Major Performance Variable #
One of the clearest themes in MLPerf Inference v6.1 is that performance improvements are increasingly coming from software as well as new silicon.
MLCommons reported that many of the strongest gains in this round came from new hardware and software/algorithmic optimization, with particularly large improvements in workloads such as VLM and DeepSeek-R1.
AMD highlighted additional performance from the same MI355X hardware after another cycle of ROCm development. NVIDIA reported that some v6.1 results were as much as 1.6x higher than v6.0, with software optimization accounting for a significant portion of the improvement.
Intel likewise attributed improvements on both Arc Pro and Xeon to continued software-stack development.
This trend has important implications for AI infrastructure evaluation.
A processor or accelerator’s practical inference performance is determined by the complete stack:
- Hardware architecture
- Memory subsystem
- Interconnect
- Compiler
- Kernel implementations
- Runtime
- Model optimization
- Quantization
- Scheduling
- Networking
- Distributed inference software
Consequently, benchmark results can change substantially even when the underlying hardware remains unchanged.
📊 What MLPerf Inference v6.1 Reveals About AI Infrastructure #
MLPerf Inference v6.1 provides several useful signals about the current direction of AI inference.
First, scale-out inference is becoming increasingly important. The record 512-GPU submission and 16 multi-node submissions demonstrate continued movement toward distributed inference systems.
Second, benchmark workloads are becoming more representative of production AI systems. End-to-end RAG and agentic inference evaluate multi-step workloads rather than isolated model execution.
Third, hardware diversity is increasing. The round spans data-center accelerators, workstation GPUs, CPUs, edge devices, and next-generation rack-scale systems.
Finally, software optimization is becoming inseparable from hardware performance. Improvements in inference engines, kernels, compilers, runtimes, and model-specific optimizations can materially change the performance of existing hardware.
🔍 Reading the Results in Context #
MLPerf Inference v6.1 does not establish a single universal performance leader across all AI inference workloads.
The results instead show different strengths at different deployment scales and workload types. AMD demonstrated the ability to scale MI355X systems to a record 512 accelerators, NVIDIA’s Blackwell systems remained highly competitive in mainstream configurations, and Vera Rubin provided the first peer-reviewed results for NVIDIA’s next-generation platform. Intel expanded its presence across both workstation GPUs and server CPUs.
More importantly, the benchmark shows that AI inference is becoming a systems-engineering problem.
Accelerator architecture remains critical, but so do memory capacity, networking, distributed execution, model optimization, software maturity, and workload design.
MLPerf results are also point-in-time measurements under specific models, configurations, software stacks, and benchmark rules. As inference frameworks, compilers, drivers, kernels, and model implementations continue to evolve, the practical performance of each platform can change substantially between benchmark generations.
For AI infrastructure teams, the most useful approach is therefore to evaluate MLPerf results against the specific model, latency target, deployment scale, hardware configuration, and software stack relevant to the intended production workload.