Google TPU 8tx and 8i Max: The Economics of Agentic Inference

Google TPU 8tx and 8i Max: The Economics of Agentic Inference

When a coding agent receives a task, it rarely generates one uninterrupted stream of tokens. It reads source files, examines documentation, runs tests, waits for tool responses, and resumes its reasoning. A workflow that looks like one continuous interaction to the user is a sequence of very different operations inside the data center.

Some stages need fast prompt processing. Others depend on low-latency token generation. During tool calls, the model may sit idle while its accumulated context remains in memory. If the task involves proprietary source code or regulated records, the infrastructure must also protect that information while the model is actively processing it.

On November 9, 2026, Google expanded its eighth-generation Tensor Processing Unit (TPU) lineup with two systems: TPU 8tx, designed for confidential inference, and TPU 8i Max, designed for large-context workloads and higher compute density. Google also introduced TPU Sync, software for moving model state across accelerator memory, host memory, and storage.

These announcements point to a broader shift in accelerator economics. The cost of agentic inference is not determined by compute throughput alone. Memory capacity, data movement, session scheduling, security requirements, and the time spent waiting for external tools all affect how much useful work a system delivers per dollar.

🧭 Why Does One TPU Generation Need Four Systems?

#

Training and inference place different demands on accelerator infrastructure.

Large-scale training requires thousands of chips to exchange data efficiently while executing coordinated workloads. Inference has a different operating profile: it must handle variable request volumes, meet latency targets, and allocate resources across workloads with different context lengths and concurrency levels.

Even inference is not one uniform problem:

• Short requests at high concurrency place pressure on scheduling and aggregate throughput.
• Long-context requests require substantial memory capacity and bandwidth.
• Real-time reasoning depends on low-latency communication and fast token generation.
• Confidential workloads require protection for data and model weights while computation is underway.
• Agentic workloads introduce idle periods when sessions wait for tools, databases, or human approval.

Google’s eighth-generation portfolio now addresses these requirements with four systems:

Google introduced the 8t and 8i at Cloud Next in April 2026. The company reported up to 2.7 times better training performance for 8t and up to 80% better inference price-performance for 8i compared with the previous generation. Those are Google’s published comparisons, not guarantees across every model or deployment.

The two new systems are not entirely new silicon designs. Google is adjusting the system around existing accelerator silicon—changing host processors, networking, and packaging to target different bottlenecks. This modular approach can bring specialized systems to market faster than developing and validating a completely new chip for every workload.

🔐 TPU 8tx Extends Protection into Active Computation

#

Encrypting data at rest and in transit is standard practice. The harder problem appears when a model needs to process that data.

A model serving private contracts, medical records, financial information, or proprietary source code must access the underlying information in memory. At that point, ordinary storage and network encryption are not enough to protect data from every threat in the execution environment.

Confidential computing aims to protect data while it is in use by establishing hardware-enforced isolation boundaries.

Google’s TPU 8tx uses the same TPU silicon as the training-oriented 8t and pairs it with fifth-generation AMD EPYC host processors. Google says its hardware-level confidential computing capabilities protect sensitive data and model weights during active processing.

That makes the system relevant to organizations that want to run proprietary models alongside sensitive enterprise data without moving those workloads into a separate, less restricted environment.

The design also supports model customization. Google describes fine-tuning, distillation, and reinforcement learning as use cases that can run on the same system, reducing the need to move sensitive data between separate clusters.

Boardfly networking targets inference latency

#

The 8tx also uses Google’s Boardfly network, which was developed for the low-latency reasoning system.

Networking topology matters because distributed inference can involve frequent communication between accelerators. This is especially relevant for Mixture-of-Experts (MoE) models, where tokens are routed to different expert networks that may reside on different chips.

Every additional communication hop can add delay. A topology designed to reduce communication latency can therefore improve the time spent waiting for remote accelerator work to complete.

Google says the 8tx delivers up to 2x lower latency than the 8t, with a near-zero performance penalty from its confidential-computing capabilities. This is a vendor-reported system comparison; actual results depend on workload, configuration, and the baseline being measured.

The practical point is that confidential computing does not have to be treated as a separate, isolated security appliance. Google is positioning the 8tx as a system that combines accelerator performance, low-latency networking, and in-use data protection.

🧠 TPU 8i Max Combines Two Chips to Address the Memory Wall

#

Long-context inference creates a memory problem that becomes more severe as sessions and concurrent users accumulate.

During inference, a transformer model stores attention keys and values from previously processed tokens in the KV cache. When the model generates the next token, it can reuse those stored values rather than recomputing the entire preceding context.

For conventional full-attention workloads, KV-cache requirements generally grow with context length. More concurrent sessions multiply that demand. The cache can consume a substantial share of accelerator memory, limiting how many sessions a system can serve at once.

Two hardware properties matter:

• Memory capacity determines how much model state and how many session caches can remain resident.
• Memory bandwidth determines how quickly that data can be supplied to the compute units.

The TPU 8i Max combines two TPU 8i chips into a single package. Google says the package provides twice the compute, twice the on-chip SRAM, and twice the HBM bandwidth of a standard 8i.

This is aimed at workloads where memory traffic can leave compute units waiting for data. Increasing bandwidth and on-chip resources helps keep the processing pipeline supplied, while the dual-chip package increases compute density within the existing rack footprint.

There is an important distinction between bandwidth and capacity. Google’s announcement specifies the doubling of compute, SRAM, and HBM bandwidth, but does not provide a complete set of absolute memory-capacity figures in the announcement. Claims about total HBM capacity should therefore be treated separately from the confirmed package-level comparisons.

Nor does a larger package automatically guarantee a particular context length. Usable context depends on model weights, attention architecture, KV-cache precision, cache layout, parallelism, and the memory reserved for other runtime operations.

For workloads involving many long-running agents or large shared codebases, the 8i Max is designed to provide more compute and memory bandwidth in a denser deployment.

🔄 TPU Sync: What Happens to Model State While an Agent Waits?

#

Consider a coding agent that reads a repository, develops a patch, and launches a test suite. The test process may take tens of seconds or several minutes. During that interval, the model is not generating tokens, but its KV cache may still occupy valuable HBM.

There are two obvious but unattractive choices:

1. Keep the entire cache in accelerator memory, reserving capacity for a session that is temporarily idle.
2. Discard the cache, forcing the system to process the earlier context again when the agent resumes.

Google’s TPU Sync is designed to provide another option: move model state between levels of the memory hierarchy according to how urgently it is needed.

The software coordinates several resources:

• Accelerator HBM
• Host DRAM
• Local or shared storage
• Data-center networking

Frequently used cache blocks can remain in HBM, while less active blocks are moved to host memory or colder storage. When a session resumes, the required state can be transferred back.

This turns memory management into a scheduling problem. The system must decide which state to retain, which state to move, and when to restore it without creating more delay than the offloading saves.

Moving KV cache between prefill and decode

#

TPU Sync also targets disaggregated inference architectures.

Inference is commonly divided into two stages:

• Prefill: Processes the input prompt and builds the initial KV cache.
• Decode: Generates output tokens incrementally using that cache.

These stages have different resource profiles. Prefill is compute-intensive, while decode repeatedly accesses model weights and cached attention state. Operators may place them on separate accelerator groups to allocate resources more efficiently.

The trade-off is data movement. When prefill and decode run on different nodes, the KV cache must be transferred between them. If that transfer takes too long, the latency benefit of separating the stages can disappear.

Google’s published TPU Sync tests include a Qwen3 235B configuration where median KV-cache transfer latency fell from several seconds to approximately 230 milliseconds. In a low-concurrency test with the same model, average Time to First Token (TTFT) decreased by about 60%.

These figures describe different measurements. KV-transfer latency measures the cache handoff itself; TTFT measures the time until the first output token arrives. A faster handoff can improve TTFT, but it is only one part of the request’s total latency.

There is another qualification: the published benchmark results were obtained on Google’s earlier TPU v7x platform, not the newly announced 8i Max or 8tx. They demonstrate the software’s potential, but should not be presented as measured performance for the new hardware.

Cache offloading is not the same as agent-state management

#

A KV cache preserves intermediate model computation. It does not, by itself, preserve the complete state of an agent application.

The orchestration layer still needs to manage task progress, tool outputs, permissions, external side effects, and retry behavior. A restored model cache does not prove that a database operation completed or that a tool call is safe to repeat.

This distinction matters in production. Model-state storage and application-state persistence are related, but they solve different problems.

For a paused session, the economic question is whether the HBM capacity released during the wait can be used by other requests—and whether restoring the cache costs less than recomputing the context.

☁️ Google Is Selling Cloud Capacity and Physical TPU Systems

#

Google’s TPU business is no longer limited to running internal workloads or renting accelerator capacity through Google Cloud.

The company describes two external delivery paths:

1. Google Cloud: Customers consume TPU compute as a cloud service.
2. Customer data centers: Organizations deploy physical TPU systems in their own facilities.

Alphabet said it began shipping TPU systems to customer data centers in the second quarter of 2026. Reuters reported that Google Cloud revenue reached approximately $30.7 billion in the third quarter, up more than 100% year over year, with TPU system sales contributing to growth. Alphabet does not separately disclose TPU hardware revenue, so the exact contribution cannot be isolated from the reported cloud figures.

The distinction between cloud capacity and physical system sales matters commercially. Cloud customers can scale usage without operating the hardware themselves. Direct system deployments give large customers more control over where accelerator infrastructure runs, which can be relevant for data residency, capacity planning, and integration with existing facilities.

Google also continues to offer NVIDIA GPUs through Google Cloud. This gives customers room to divide workloads according to model compatibility, software requirements, and measured economics rather than committing to one accelerator architecture for everything.

For the newly announced 8tx and 8i Max, Google has not yet published complete pricing or regional availability details. A meaningful purchasing comparison will require those figures, along with system-level performance data under representative workloads.

🧰 Software Determines How Much Work Can Move to TPUs

#

Hardware specifications matter only if the software stack can use them effectively.

Specialized accelerators have historically required developers to work with specific compilers, frameworks, and optimized execution paths. That creates migration costs for teams whose models and serving infrastructure already run on established GPU stacks.

Google is trying to reduce that barrier through broader framework support and tools for operating TPU workloads.

TorchTPU, vLLM, and SGLang

#

Google has open-sourced TorchTPU, a PyTorch backend intended to let developers run PyTorch applications and Hugging Face models natively on TPUs.

Google is also working with established inference engines, including vLLM and SGLang, to provide TPU support.

This matters because framework compatibility affects the real cost of migration. A team evaluating a new accelerator must account for more than the time required to port model code. It also needs to verify operator coverage, quantization support, custom kernels, distributed execution, debugging tools, and production stability.

A familiar API does not guarantee identical performance or complete compatibility. Each model and serving configuration still needs to be tested.

Tools for TPU optimization

#

Google has also introduced a TPU agent suite in preview. It is designed to work with coding tools such as Claude Code, Cursor, and Google Antigravity to assist with environment setup, performance diagnosis, and kernel optimization.

For engineering teams, the goal is to reduce the amount of hardware-specific knowledge required to get a workload running efficiently.

Google’s XLA Serving Engine is another part of the software effort. It is designed to provide a low-latency C++ serving runtime for models developed in Python, across TPU, GPU, and CPU environments.

These projects address a familiar infrastructure problem: an accelerator can have attractive theoretical performance, but if developers cannot deploy and maintain workloads on it without substantial engineering effort, its practical value is lower.

⚙️ GKE and Cloud Pathways Improve Cluster Utilization

#

Operating a large accelerator fleet introduces another source of cost: resources that are provisioned but not doing useful work.

Google’s infrastructure announcements include improvements to distributed execution and TPU allocation through Google Kubernetes Engine (GKE).

Cloud Pathways

#

Cloud Pathways brings Google’s distributed runtime for large-scale AI workloads to Google Cloud customers. It uses a single-controller architecture to coordinate TPU workloads across multiple slices.

The runtime is intended for pre-training, reinforcement learning, and multi-host inference. Google describes support for long-running jobs through mechanisms such as colocated Python sidecars, slice hot-swapping, and elastic resharding.

These features address operational interruptions. If a hardware slice becomes unavailable, a distributed job may otherwise lose time waiting for resources to be restored or for the workload to be restarted.

Google says Cloud Pathways is generally available on GKE.

GKE dynamic slicing

#

Traditional accelerator deployments often rely on predefined resource pools. Changing the allocation can require infrastructure operations that take time and leave hardware underused.

GKE dynamic slicing uses optical circuit switching (OCS) to separate physical hardware provisioning from logical TPU slice formation. Operators can assemble resources into larger configurations or divide them into smaller isolated workloads as requirements change.

Google describes two operations:

• Super-slicing: Combines TPU resources into larger logical topologies.
• Sub-slicing: Divides available resources into smaller, isolated configurations.

Google reports that dynamic slicing can reduce job startup time by up to 5x and improve fault recovery by up to 4.5x. These are vendor-reported results, and the actual benefit will depend on cluster configuration and workload behavior.

The underlying economic objective is straightforward: keep expensive accelerator capacity allocated to useful work instead of tying it up in rigid configurations or lengthy recovery procedures.

📊 The Real Metric Is Cost per Completed Agent Task

#

For a conventional inference service, comparing accelerators often starts with throughput, latency, and cost per million tokens. That remains useful, but agentic workloads introduce additional costs that token-level benchmarks can miss.

An agent may generate a few hundred tokens, wait for a test suite, resume with a large context, call another tool, and repeat the process. Its total infrastructure cost includes both active computation and the resources reserved while it is idle.

A more useful comparison should account for:

The relevant metric becomes total cost per successfully completed task, not simply cost per generated token.

For example, a system with higher peak token throughput may still be more expensive for agentic workloads if it cannot release memory during long pauses. A slower accelerator may be more economical if its cache management and scheduling allow substantially more concurrent sessions.

The same applies to security. A confidential-computing system may carry a premium, but that cost could be justified if it allows sensitive workloads to run without separate infrastructure or complicated data movement.

Google’s latest TPU systems address different pieces of this equation. The 8tx targets in-use data protection and low-latency communication. The 8i Max targets memory bandwidth and compute density. TPU Sync addresses the movement and reuse of model state, while the broader software stack aims to reduce migration and orchestration costs.

The architectural direction is clear: agentic inference is becoming a system-level optimization problem. The accelerator matters, but so do the memory hierarchy, network, scheduler, security boundary, and software that determine how much useful work the entire platform completes.