Do We Still Need GPUs? How AI-Accelerated CPUs Could Change HPC
For more than a decade, GPUs have been the dominant hardware platform for artificial intelligence and high-performance computing (HPC). Large language model (LLM) training, scientific simulations, climate modeling, and materials research have increasingly depended on GPU acceleration.
But that dominance is facing a fundamental question: do we still need GPUs for every AI and HPC workload?
A landmark paper, Do We Still Need GPUs?, co-authored by supercomputing researchers including Jack Dongarra, Torsten Hoefler of ETH Zurich, and Satoshi Matsuoka of RIKEN, explores how advances in CPU architecture could change the balance.
The central argument is not that GPUs are becoming obsolete. Instead, increasingly capable CPUs equipped with matrix engines, vector extensions, high-bandwidth memory (HBM), and broad multi-precision support could handle a much larger share of AI and scientific workloads.
For frontier-scale model training, GPUs are likely to remain essential in the near term. But inference, fine-tuning, scientific machine learning, and tightly coupled simulation workloads could increasingly migrate toward specialized CPUs.
⚡ Why GPUs Became the Default for AI and HPC #
GPUs were originally developed for graphics rendering, where thousands of operations could be executed in parallel. Researchers later recognized that this massively parallel architecture was also well suited to scientific and AI workloads.
Three characteristics drove GPU adoption:
- High compute throughput: GPUs can execute enormous numbers of floating-point and matrix operations in parallel.
- Massive memory bandwidth: High-bandwidth memory keeps thousands of compute units supplied with data.
- AI-specific acceleration: Tensor cores and low-precision formats dramatically improve deep-learning performance.
The surrounding software ecosystem reinforced this advantage. AI frameworks, scientific libraries, and distributed computing tools increasingly optimized their execution paths around GPUs.
However, GPU dominance was also partly a consequence of historical CPU limitations. Traditional CPUs offered fewer arithmetic units, narrower memory bandwidth, and less specialized support for low-precision matrix operations.
Modern CPUs are changing that equation.
As CPUs acquire many of the capabilities traditionally associated with accelerators, the distinction between a “CPU workload” and a “GPU workload” becomes increasingly dependent on compute intensity, memory bandwidth, data movement, and programmability rather than processor category alone.
🧠 Modern CPUs Are Adopting GPU-Like Capabilities #
Next-generation CPUs increasingly integrate hardware that was once considered almost exclusively the domain of GPUs.
Examples include:
- ARM SVE and SME vector and matrix extensions.
- Intel AVX and AMX instruction sets.
- Integrated or in-package HBM.
- Hardware support for BF16, INT8, FP8, FP4, and other precision formats.
- Dedicated matrix-multiplication engines.
These features allow CPUs to address both traditional control-heavy workloads and increasingly demanding AI calculations.
Eliminating Data-Movement Bottlenecks #
One of the strongest arguments for CPU-based AI acceleration is the elimination of unnecessary data movement.
Traditional heterogeneous systems frequently move data between CPU memory and GPU memory. These transfers consume both time and energy and can complicate software design.
A CPU with integrated matrix acceleration and high-bandwidth memory can operate within a unified memory environment.
This can be particularly valuable for workloads that repeatedly exchange data between AI models and conventional scientific or application logic.
Combining AI and Scientific Computing #
AI workloads often favor dense matrix multiplication, while scientific applications can contain:
- Sparse matrices.
- Irregular memory access.
- Conditional branching.
- Adaptive meshes.
- Complex control flow.
GPUs excel at highly regular parallel computation but can become less efficient when workloads contain substantial irregularity.
An enhanced CPU can combine matrix acceleration with conventional CPU execution resources, allowing the same processor to handle both AI tensor operations and complicated scientific code.
This makes CPU-centric architectures attractive for hybrid AI-and-simulation workloads.
Supporting Multiple Precision Levels #
AI and scientific computing have very different numerical requirements.
LLM inference frequently benefits from BF16, INT8, FP8, or FP4, while many scientific simulations depend on FP64 for numerical accuracy.
A CPU equipped with flexible matrix hardware can potentially support this broader precision spectrum within one architecture.
That opens the door to mixed-precision applications such as:
- Quantized LLM inference.
- AI-assisted physics simulations.
- Scientific surrogate models.
- Machine-learning-based numerical solvers.
Building a More Balanced System #
Peak FLOPS alone does not determine application performance.
A practical accelerator must balance:
- Compute throughput.
- Memory bandwidth.
- Memory capacity.
- Interconnect performance.
- Power consumption.
- Software complexity.
This is an area where enhanced CPUs can be compelling. Instead of maximizing one particular metric, they can provide a more balanced computing platform.
🚀 Where CPUs Could Challenge GPUs #
Not every AI workload requires the extreme parallelism of a large GPU cluster.
Matrix-accelerated CPUs are particularly interesting for several categories.
LLM Inference #
LLM inference is often constrained by memory bandwidth, especially during token generation.
The processor must repeatedly stream model weights and KV-cache data from memory. In such situations, adding more arithmetic throughput does not necessarily improve performance if the compute units are already waiting for memory.
High-bandwidth CPU architectures can therefore be surprisingly competitive.
Fine-Tuning and Smaller AI Models #
Fine-tuning workloads generally require less computational capacity than training a frontier model from scratch.
For organizations that already operate large CPU infrastructure, using matrix-accelerated CPUs could reduce the need for separate GPU resources for certain AI workloads.
Scientific Machine Learning #
Scientific ML often combines neural networks with conventional numerical algorithms.
Because these workloads can alternate between dense AI operations and irregular scientific computation, CPUs with integrated matrix acceleration may provide a better balance than a discrete accelerator.
Hybrid Simulation and AI #
One particularly interesting opportunity involves simulations where AI models are invoked repeatedly inside a larger physics or engineering workflow.
Keeping both workloads on the same processor can reduce the overhead of transferring data between CPU and GPU memory.
🔥 Where GPUs Still Have a Major Advantage #
Despite CPU advances, the paper does not suggest that GPUs are about to disappear.
The strongest remaining GPU advantage is frontier-scale AI training.
Training enormous models requires more than matrix performance from an individual processor. It requires an entire system consisting of:
- Extremely high compute density.
- Massive memory bandwidth.
- High-capacity accelerator memory.
- High-speed accelerator-to-accelerator interconnects.
- Efficient distributed training software.
- Mature AI frameworks and libraries.
- Strong power efficiency at cluster scale.
Replicating this complete ecosystem with CPUs is considerably more difficult than simply adding matrix instructions.
For training the largest frontier models, GPUs therefore remain the practical choice for the foreseeable future.
📊 ARM CPU Experiments Provide an Interesting Comparison #
The researchers examined these concepts using large-scale LLM inference workloads, including trillion-parameter Mixture-of-Experts (MoE) models with long context windows.
Two ARM-based platforms provide an interesting contrast:
- Fugaku A64FX: Wide vector units and HBM, but without a dedicated matrix engine.
- LX2: Combines vector processing, integrated SME matrix engines, and in-package HBM.
The comparison highlights an important distinction between the two major phases of LLM inference.
Decode Is Primarily Memory-Bound #
During token generation, model weights and the KV cache must continually be streamed from memory.
On the A64FX platform, roughly 80% of execution time was attributed to waiting for HBM data transfers, while floating-point computation accounted for less than 1%.
This demonstrates that simply increasing compute throughput does little when memory bandwidth is the limiting factor.
The results also showed that decode performance scaled closely with available memory bandwidth.
A sufficiently large CPU cluster with enough HBM bandwidth can therefore approach GPU-class performance on certain memory-bound inference workloads.
Prefill Is Primarily Compute-Bound #
The situation changes during the prefill phase.
Processing a large prompt requires substantial dense matrix computation, making specialized matrix engines much more important.
The A64FX, which lacks a dedicated matrix accelerator, fell dramatically behind modern GPUs in this phase.
The analysis indicates that newer CPU designs with substantial BF16 and INT8 matrix throughput can cross the performance threshold needed for practical prefill workloads, particularly when combined with efficient attention techniques.
This distinction is crucial: the best processor depends on which part of the AI workload is actually limiting performance.
⚠️ The Remaining Challenges for CPU-Centric AI #
CPU-based acceleration is promising, but several major obstacles remain.
1. Interconnect Scaling #
Training trillion-parameter models requires communication across hundreds or thousands of processors.
Tensor parallelism and pipeline parallelism can generate enormous communication requirements.
GPU platforms have invested heavily in specialized high-bandwidth interconnects, making them difficult for conventional CPU clusters to match.
2. Power Efficiency #
CPU systems using older generations of HBM may consume substantially more energy per user or inference request than GPU systems equipped with newer HBM technologies.
This is especially important at data-center scale, where electricity and cooling can become major operational constraints.
Future CPU generations will need improvements in both compute efficiency and memory efficiency to compete directly with GPUs.
3. Low-Precision Hardware #
Modern AI increasingly depends on very low-precision arithmetic.
First-generation matrix-enhanced CPUs may lack native support for some formats such as FP8, forcing workloads to rely on BF16 or INT8.
As CPU matrix extensions evolve, broader hardware support for emerging AI precision formats will become increasingly important.
4. Software Ecosystem #
Hardware performance is only one part of the equation.
GPU platforms benefit from mature AI libraries, compilers, distributed-training frameworks, optimized kernels, and extensive developer experience.
CPU architectures need comparable software optimization before their theoretical hardware capabilities translate into widespread application-level performance.
🔮 So, Can CPUs Replace GPUs? #
The answer is sometimes—but not universally.
GPUs are likely to remain the primary accelerators for training the largest frontier AI models because of their combination of compute density, memory bandwidth, interconnect performance, energy efficiency, and mature software ecosystems.
However, CPUs are becoming increasingly capable AI accelerators in their own right.
For workloads such as:
- LLM inference.
- Model fine-tuning.
- Scientific machine learning.
- Graph neural networks.
- AI-assisted simulations.
- Hybrid physics-and-AI workflows.
matrix-enhanced CPUs could become increasingly attractive.
The most important development may therefore not be the replacement of GPUs, but the emergence of a more heterogeneous computing landscape in which CPUs and GPUs each handle the workloads they are best suited for.
🌐 The Future May Be About Convergence, Not Replacement #
The boundary between CPUs and accelerators is steadily becoming less distinct.
CPUs are gaining:
- Vector engines.
- Matrix accelerators.
- HBM.
- Low-precision arithmetic.
- AI-specific instructions.
- Higher memory bandwidth.
Meanwhile, GPUs continue expanding their capabilities in general-purpose computing, memory management, and increasingly complex control workloads.
Rather than asking whether CPUs or GPUs will win, the more useful question may be:
Which architecture provides the best balance of compute, memory, communication, power, and programmability for a particular workload?
For frontier model training, GPUs currently have the advantage.
For memory-bound inference and tightly integrated scientific workloads, increasingly powerful CPUs could become serious competitors.
The long-term result may be a convergence of processor architectures in which the traditional distinction between “CPU” and “GPU” becomes less important than the capabilities integrated into the computing system.
The GPU era is far from over. But the rise of AI-accelerated CPUs suggests that the future of AI computing may contain far more CPU acceleration than the industry once expected.