Why Memory Bandwidth, Not Compute, Determines LLM Inference Speed

Sat, 16 May 2026 12:53:51 +0800

Why Memory Bandwidth, Not Compute, Determines LLM Inference Speed

In an in-depth discussion with Dwarkesh Patel, Reiner Pope, MatX founder and former Google TPU architect, sheds light on why stacking raw compute power (FLOPS) fails to reduce single-user LLM latency. He explores memory bandwidth limits, Mixture of Experts (MoE) routing challenges, KV cache management, and the economic implications for API pricing in 2026.

Mixture of Experts on KAD

Why Memory Bandwidth, Not Compute, Determines LLM Inference Speed