Executive Summary

A Tensor Processing Unit (TPU) is a custom application-specific integrated circuit (ASIC) designed by Google specifically for the matrix arithmetic at the foundation of modern deep neural network workloads. Google has deployed seven production generations since 2015 and announced an eighth generation that structurally bifurcates training and inference into distinct physical chips. This report analyzes how a TPU executes workloads, what microarchitectural changes emerged across major generations, how thousands of chips assemble into unified pods via optical circuit switches, and how the architecture fundamentally compares with contemporary GPUs.

Compute Core • MXU
Systolic Matrix Engine
256 × 256 MAC Grid (v6e / TPU7x)
MACs per MXU 65,536 Cells
Ironwood BF16 / FP8 2,307 / 4,614 TFLOPs
On-Chip Memory Software VMEM (No L1/L2)
Dispatch Model Static AOT XLA VLIW
Supercomputer • Pod
3D Torus + Optical OCS
4 × 4 × 4 Copper Cubes + Photonic Switches
Max Pod Scale 9,216 Chips (Ironwood)
Per-Chip ICI Bandwidth 1,200 GB/s (6 Links)
HBM3e per Socket 192 GiB @ 7.37 TB/s
Fault Tolerance MEMS Mirror Bypass
8th Gen • 2026*
TPU 8t & TPU 8i Split
Dedicated Training vs Inference Silicon
TPU 8t (Training) Native FP4 • 9,600 Pod
TPU 8t ICI Fabric 19.2 Tb/s 3D Torus
TPU 8i (Inference) 288 GB HBM + 384 MB SRAM
TPU 8i Topology Boardfly (7-Hop Max)

Key Findings

  • Design: A TPU dedicates the vast majority of its silicon real estate to a small number of large systolic-array matrix multiply units (MXUs), fed directly by software-managed on-chip SRAM scratchpad memory (VMEM) and scheduled deterministically ahead of time by the XLA compiler. It deliberately omits the hardware warp schedulers, dynamic dispatch units, and multi-level cache hierarchies found in GPUs [1], [10].
  • Performance Trajectory: Peak BF16 throughput per chip grew from 123 TFLOPs on TPU v3 (2018) to 2,307 TFLOPs on Ironwood (2025)—an $18.8\times$ increase per chip, further reaching 4,614 TFLOPs in native FP8 [3], [8].
  • Scale: An Ironwood pod interconnects up to 9,216 chips as a single coherent scale-up domain through a 3D torus constructed from $4 \times 4 \times 4$ cubes, with reconfigurable Optical Circuit Switches (OCS) joining the cubes dynamically [1], [8].
  • Hardware Specialisation: The eighth generation (TPU 8t and 8i) separates training from inference, optimizing precision (native FP4 on 8t), memory hierarchies (384 MB on-chip SRAM on 8i for KV caching), and network topologies [10].
  • Comparison with GPUs: The most decisive differences reside at system scale: a single TPU scale-up domain spans thousands of direct-routed chips rather than tens of GPUs behind InfiniBand/RoCE fabrics. Published figures often conflate sparse vs. dense numbers and unidirectional vs. bidirectional interconnect bandwidth [4].

Basis of the Figures

Unless explicitly noted otherwise, specifications derive from primary Google Cloud architecture documentation [1]–[9] and peer-reviewed IEEE/ACM ISCA publications. Technical details for TPU 8t and 8i originate from primary Google technical disclosures and analytical industry reporting [10]. All specifications reflect verified status as of November 2026.

1. How a TPU Works

1.1 Systolic Design Principle

The computational heart of a TPU is a systolic array: a physical 2D grid of multiply-accumulate (MAC) units hardwired directly to their orthogonal neighbors. In a traditional processor, intermediate matrix calculations must continually write back to registers or cache memory. In a systolic array, data streams continuously across the grid, handing intermediate partial products directly to adjacent accumulators. Matrix multiplication completes without intermediate memory traffic [1].

Table 1. Matrix multiplication execution patterns across computing architectures
Processor Execution Pattern & Bottleneck Analysis
CPU Loads operands from L1/L2 caches into register files, executes ALU instructions, and writes back for every operation. Arithmetic intensity is low; memory bus bandwidth caps throughput (the von Neumann bottleneck).
GPU Thousands of lightweight SIMT ALUs work in parallel across Streaming Multiprocessors (SMs). Intermediate results cycle through register files or shared memory banks. Hardware warp schedulers hide memory latency through aggressive multi-threading.
TPU Thousands of multiply-accumulators form a continuous physical systolic grid. Stationary weights are loaded into cell registers, activations stream through the array, and accumulated products pass downstream. No intermediate memory access occurs during the core dot-product calculation.
Source: Synthesized from Google Cloud TPU architecture documentation [1] and ISCA foundational papers [11].

1.2 Core Components

A production TPU chip integrates one or more TensorCores with high-bandwidth memory (HBM), vector/scalar processing units, inter-chip interconnect links, and specialized SparseCores. Table 2 details the principal functional units and Figure 1 illustrates how data moves across the hierarchy.

Table 2. Principal components of a TPU chip
Component Microarchitectural Function & Dimensionality
TensorCore The primary compute core containing the systolic Matrix Multiply Units (MXUs), a wide vector unit, a scalar control unit, and dedicated Vector Memory (VMEM) scratchpad.
MXU (Matrix Multiply Unit) The systolic compute engine. Dimensioned at $128 \times 128$ up to TPU v5p (16,384 MACs) and scaled to $256 \times 256$ on v6e (Trillium) and TPU7x (Ironwood), quadrupling density to 65,536 MACs per unit.
Precision Engine Inputs process as Bfloat16, with lower-precision native formats on modern silicon (INT8 on v5e/v6e; native FP8 on v5p/Ironwood; native FP4 on 8t). Accumulation is strictly maintained in FP32.
Vector & Scalar Units Vector units compute elementwise activations (GELU, SwiGLU), LayerNorm/RMSNorm, and Softmax reductions. The scalar unit manages loop branching, instruction dispatch, and address generation.
HBM & DMA Subsystem High-Bandwidth Memory stacks store global model weights, optimizer states, and KV-cache tensors. Data streams to on-chip VMEM scratchpads through asynchronous hardware DMA queues (infeed/outfeed).
SparseCore A specialized asynchronous dataflow coprocessor engineered to accelerate sparse, memory-bandwidth-bound embedding table lookups in large recommendation models (DLRMs). v5p and TPU7x integrate four per chip; v6e integrates two.
Source: Google Cloud TPU documentation [1], [6]–[8].
Figure 1: TensorCore Internal Data Path & Systolic Hierarchy
TENSORCORE PROCESSING HIERARCHY Host CPU PCIe Gen5 / CXL Infeed Queue HBM3 / HBM3e Weights & States Up to 7.37 TB/s 192 GiB (Ironwood) On-Chip VMEM Software Scratchpad Zero Cache Hierarchy DMA Staging Engine Tiles & Ring Buffers MXU (Systolic Array) 256 × 256 Multiply-Accumulate 65,536 MACs • Bfloat16 / FP8 / FP4 Stationary Weights • FP32 Acc Vector & Scalar Units Softmax • GELU • RMSNorm • Transpose VLIW Instruction Stream • Address Gen
Figure 1. Simplified TensorCore data path. Host inputs queue into high-bandwidth memory (HBM). Software-managed DMA controllers stream 2D tiles into the on-chip Vector Memory (VMEM) scratchpad without dynamic cache tagging. Stationary weights initialize the $256 \times 256$ systolic array, while activations pass orthogonally through MAC cells with FP32 accumulation. Vector and scalar units compute non-GEMM activations, normalizations, and address arithmetic.

1.3 The Role of the XLA Compiler

A TPU offloads control-flow complexity from silicon to software. Unlike GPUs, which expend significant die area on hardware warp schedulers, dynamic scoreboard trackers, and out-of-order execution logic, a TPU operates deterministically under the ahead-of-time (AOT) control of Google's XLA (Accelerated Linear Algebra) compiler [10].

XLA ingests computational graphs from high-level frameworks (JAX, PyTorch via PyTorch-XLA), decomposes operations into primitives, performs aggressive operator fusion, and directly schedules every cycle of computation, DMA transport, and inter-chip collective communication. Because execution timing is completely static and cycle-accurate, thousands of chips remain lockstep-synchronized during distributed all-reduce operations without incurring runtime communication jitter.

The Padding Penalty & Ragged Attention

The primary trade-off of a massive fixed-dimension systolic array is matrix dimension mismatch. When operational matrix dimensions are not integer multiples of the systolic grid ($128$ or $256$), XLA must pad dimensions with zeros. Unpadded irregular token counts or dynamic sequence lengths cause execution lanes to sit idle. Addressing this limitation motivated ragged paged-attention algorithms engineered specifically to pack irregular LLM decode batches into dense, tile-aligned systolic blocks [12].

2. How the Major Generations Differ

Google's architectural progression reflects an initial focus on internal recommendation and vision models, transitioning to massive distributed LLM pre-training, and culminating in specialized dual-silicon systems.

Figure 2: Peak Dense BF16 Compute Progression per Chip (TFLOPs)
0 500 1,000 1,500 2,000 2,500 123 TPU v3 (2018) 275 TPU v4 (2021) 197 TPU v5e (2023) 459 TPU v5p (2023) 918 v6e (Trillium) (2024) 2,307 TPU7x (Ironwood) (2025) 4,600+* TPU 8t (2026*)
Figure 2. Generational progression of peak BF16 dense compute per chip from TPU v3 (2018) to Ironwood (2025) and TPU 8t (2026). Ironwood achieves 2,307 BF16 TFLOPs (and 4,614 FP8 TFLOPs) through dual chiplets and $256 \times 256$ arrays—representing an $18.8\times$ scaling over v3. (*TPU 8t reflects preliminary architectural disclosures utilizing native FP4/BF16 tensor engines).
Table 3. Comprehensive microarchitectural comparison across TPU generations
Generation Year Compute Engine Peak Compute (TFLOPs) HBM Capacity & BW Interconnect Max Pod Size
TPU 8i [10] 2026* 2 TensorCores; Collectives Engine Serving Optimized 288 GB HBM + 384 MB SRAM Boardfly Mesh 1,152 chips
TPU 8t [10] 2026* 1 TensorCore; SparseCores; FP4 ~4,600+ BF16 / FP4 216 GB HBM 3D Torus (19.2 Tb/s) 9,600 chips
Ironwood (TPU7x) [8] 2025 Dual Chiplet (2 TC, 4 SparseCores) 2,307 BF16 / 4,614 FP8 192 GiB (7.37 TB/s) 3D Torus (1,200 GB/s) 9,216 chips
v6e (Trillium) [7] 2024 1 TC (two $256 \times 256$ MXUs), 2 SC 918 BF16 / 1,836 INT8 32 GB (1.6 TB/s) 2D Torus 256 chips
v5p [6] 2023 2 TC (four $128 \times 128$ MXUs), 4 SC 459 BF16 / 918 FP8 95 GiB (2.76 TB/s) 3D Torus with OCS 8,960 chips
v5e [5] 2023 1 TC (four $128 \times 128$ MXUs) 197 BF16 / 393 INT8 16 GB (819 GB/s) 2D Torus 256 chips
v4 [4] 2021 2 TC (four $128 \times 128$ MXUs), 4 SC 275 BF16 32 GiB (1.2 TB/s) 3D Torus with OCS 4,096 chips
v3 [3] 2018 2 TC (two $128 \times 128$ MXUs each) 123 BF16 32 GiB (900 GB/s) 2D Torus (Copper) 1,024 chips
Source: Google Cloud TPU documentation [3]–[8]; TPU 8t/8i data synthesized from StorageReview [10]. *Announced 2026.

2.1 What Changed at Each Step

v1 • 2015 The Inference Benchmark

Deployed strictly for inferencing DeepMind and Google Search workloads. Featured a single 65,536-MAC 8-bit matrix unit operating at 92 TOPS with 28 MiB of software-managed SRAM, demonstrating that deterministic execution suits tail-latency bounds far better than caches and out-of-order pipelines [11].

v2 & v3 • 2017–2018 Training Arrives & Bfloat16

Added support for backpropagation, co-invented the Brain Floating Point (bfloat16) format, integrated High-Bandwidth Memory (HBM), and linked pods over a direct-connected 2D torus scaling to 1,024 liquid-cooled v3 chips (123 BF16 TFLOPs per chip) [2], [3].

v4 • 2021 Optical Circuit Switches (OCS) & 3D Tori

Replaced rigid copper cables at cluster boundaries with dynamic optical circuit switches. Each chip integrated 6 Inter-Chip Interconnect (ICI) links, forming reconfigurable $4 \times 4 \times 4$ cubes scaling to 4,096-chip 3D tori. Introduced specialized SparseCores for recommendation embeddings [4].

v5e & v5p • 2023 Cost Efficiency vs. Flagship Scale Specialization

Split development into two parallel silicon tiers: v5e (cost-optimized serving and fine-tuning with 1 TensorCore and 2D torus in 256-chip slices) and v5p (flagship training supercomputer with 2 TensorCores, 95 GiB HBM, and 8,960-chip 3D torus scaling) [5], [6].

v6e Trillium • 2024 256 × 256 MXU Array Expansion

Quadrupled MAC density per MXU by expanding systolic arrays from $128 \times 128$ to $256 \times 256$, lifting peak BF16 compute per chip from 197 to 918 TFLOPs ($4.7\times$ the v5e). Retained high-density 256-chip 2D torus networking for serving and fine-tuning [7].

Ironwood / TPU7x • 2025 Chiplets, Native FP8 & 9,216-Node Pods

Google's first chiplet-based TPU with native FP8. Each physical socket packages two chiplets (each with its own TensorCore, two SparseCores, and 96 GiB of HBM), exposed to frameworks such as JAX as two runtime devices. Delivers 2,307 BF16 TFLOPs (4,614 FP8 TFLOPs) and 192 GiB HBM at 7.37 TB/s across 9,216-chip 3D tori at $2\times$ the performance per watt of Trillium [8], [9].

TPU 8t & 8i • 2026* Clean Separation of Training and Serving Silicon

The 8t focuses exclusively on massive pre-training with native FP4 arithmetic, 216 GB HBM, and inter-chip bandwidth doubled to 19.2 Tb/s per chip. The 8i re-engineers inference serving: it replaces SparseCores with a hardware Collectives Acceleration Engine, integrates 384 MB of on-chip SRAM to hold active KV-caches alongside 288 GB HBM, and adopts a Boardfly topology reducing worst-case hops across 1,024 chips from 16 to 7 [10].

4. TPU and GPU Compared

While both GPUs and TPUs process modern tensor workloads, they embody fundamentally contrasting systems engineering philosophies:

Table 4. Architectural comparison: TPU (Ironwood / 8t) vs. Modern GPU (NVIDIA Hopper / Blackwell)
Dimension Google TPU (Ironwood / 8-Series) NVIDIA GPU (Hopper H100 / Blackwell B200)
Compute Core Few large 2D systolic arrays ($256 \times 256$ MACs) + wide SIMD vector unit. Many Streaming Multiprocessors (SMs), each with 64/128 CUDA cores + 4 Tensor Cores.
Scheduling & Dispatch Static, ahead-of-time VLIW instruction scheduling by XLA compiler. No warp schedulers. Dynamic runtime hardware warp schedulers (32 threads per warp) hiding memory stalls.
Memory Hierarchy Flat HBM + large software-managed scratchpad SRAM (VMEM). Zero hardware L1/L2 tags. Multi-tiered hardware cache hierarchy (L1/Shared Memory, massive L2 cache, HBM).
Scale-Up Domain Direct-routed 3D Torus via ICI & OCS across 9,216 chips per pod. NVLink/NVSwitch domain typically spans 8 to 72 GPUs; wider scale requires InfiniBand/RoCE.
Network Switching Optical Circuit Switches (OCS) switching pure photons without transceivers. Electronic packet switches (NVSwitch, Quantum-2 InfiniBand, Spectrum-X Ethernet).
Execution Model Deterministic, cycle-accurate timing. Highly predictable collective latency. Asynchronous, warp-divergent execution. Highly adaptable to dynamic kernel execution.
Precision Roadmap Pioneered Bfloat16; native FP8 on Ironwood; native FP4 on TPU 8t. Pioneered TensorFloat-32 (TF32); 4th-Gen FP8 (Hopper); 2nd-Gen NV-FP4 (Blackwell).

4.1 Reading Published Comparisons

Published benchmark head-to-heads between TPUs and GPUs frequently introduce confounding variables. Engineers evaluating published claims must normalize three common discrepancies:

  • Sparse vs. Dense FLOPS: NVIDIA datasheets frequently headline Tensor Core throughput assuming 2:4 structured sparsity, which mathematically doubles theoretical peak FLOPS by assuming 50% zero-weights. TPU specifications quote strictly dense matrix throughput. A 2,307 TFLOP Ironwood chip should be compared to the dense FP16/BF16 rating of an H100 (989 TFLOPs), not its 1,979 TFLOP sparse figure.
  • Unidirectional vs. Bidirectional Interconnect: Google documentation specifies ICI bandwidth in aggregate bidirectional terms (e.g., 1,200 GB/s on Ironwood across 6 links, meaning 100 GB/s transmit + 100 GB/s receive per link direction). GPU specifications alternate between per-link bidirectional and unidirectional NVLink numbers.
  • Cluster Power & PUE: Because TPUs eliminate electronic InfiniBand switches inside the pod and replace them with passive optical circuit switches, pod-level cooling and power overhead (PUE) are significantly lower than equivalent multi-switch GPU superclusters [4].

5. Choosing a Platform and a First Program

5.1 Workload Fit

Neither architecture is universally superior. Selection depends on mathematical workload structure, software ecosystem, and cluster scale:

  • When TPU Excels: Large dense transformer training (LLMs, multimodal backbones) where sequence shapes are stable; ultra-large distributed jobs requiring synchronous all-reduce across thousands of nodes without network congestion; teams natively developing in JAX or PyTorch-XLA; workloads where deterministic cost-per-token dominates.
  • When GPU Excels: Inference serving with extreme dynamic batching and ragged sequence lengths; models relying heavily on custom hand-written CUDA/Triton kernels (e.g., non-standard attention variants, state-space models like Mamba); graph neural networks and irregular sparse architectures; teams requiring immediate local desktop or on-premise prototyping.

5.2 A First Program in JAX

JAX is the native programming model for the TPU. The following program demonstrates tensor device enumeration, ahead-of-time compilation via XLA, and multi-device 2D matrix sharding across a TPU mesh:

Distributed TPU Matrix Sharding & XLA JIT Compilation (Python / JAX)
import jax
import jax.numpy as jnp
from jax.sharding import Mesh, PartitionSpec as P, NamedSharding

# 1. Enumerate available physical TPU devices
devices = jax.devices()
print(f"Connected TPU Devices: {devices}")
print(f"Device Kind: {devices[0].device_kind}")

# 2. Form a 2D logical device mesh (e.g. data & model parallelism)
# Assuming 8 devices: 2 data-parallel axes, 4 model-parallel axes
mesh = Mesh(devices=jax.devices().reshape(2, 4), axis_names=('data', 'model'))

# 3. Define 2D sharded matrices
# Matrix X is sharded along data parallel axis; W is sharded along model axis
dim_b, dim_k, dim_out = 8192, 4096, 8192

sharding_x = NamedSharding(mesh, P('data', None))
sharding_w = NamedSharding(mesh, P(None, 'model'))

x = jax.random.normal(jax.random.PRNGKey(0), (dim_b, dim_k), dtype=jnp.bfloat16)
w = jax.random.normal(jax.random.PRNGKey(1), (dim_k, dim_out), dtype=jnp.bfloat16)

# Transfer tensors to TPU HBM with explicit sharding layouts
x_sharded = jax.device_put(x, sharding_x)
w_sharded = jax.device_put(w, sharding_w)

# 4. Ahead-Of-Time (AOT) Compiled Matrix Multiplication via XLA
@jax.jit
def parallel_matmul(a, b):
    # Systolic array executes dot-product; outputs sharded along ('data', 'model')
    return jnp.matmul(a, b)

# Execute on physical silicon
out = parallel_matmul(x_sharded, w_sharded)
out.block_until_ready()
print(f"Output Matrix Shape: {out.shape}, Sharding: {out.sharding}")

6. Conclusion

Google's Tensor Processing Unit represents an architectural thesis: hardware should be radically simplified around dense matrix arithmetic, while compiler software should bear the burden of scheduling and synchronization. By substituting complex hardware warp managers and cache hierarchies with massive systolic arrays, software-managed scratchpads, and direct optical circuit switching, the TPU achieves extraordinary performance density and scales thousands of nodes into unified, direct-routed supercomputers.

As the industry bifurcates into distinct pre-training scaling limits (demanding 10,000-node FP4 fabrics) and high-concurrency low-latency serving (demanding massive SRAM KV-caches and Boardfly topologies), the eighth-generation TPUs solidify this specialized, domain-specific trajectory.

Scope and Limitations

TPU deployments are constrained to Google Cloud and internal Google datacenters; on-premises hardware ownership is not commercially supported. Workloads that cannot be framed as dense tensor operations or that require highly dynamic control flow will encounter lower hardware utilization due to systolic padding constraints.

References

[1] Google Cloud. “TPU System Architecture Documentation.” Google Cloud Technical Docs, 2026.
[2] Google Cloud. “TPU v2 Details.” Google Cloud Technical Docs, 2026.
[3] Google Cloud. “TPU v3 Details & Hardware Specifications.” Google Cloud Technical Docs, 2026.
[4] N. P. Jouppi, et al. “TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddings.” ACM/IEEE 50th Annual International Symposium on Computer Architecture (ISCA), 2023.
[5] Google Cloud. “TPU v5e Hardware Architecture Guide.” Google Cloud Technical Docs, 2026.
[6] Google Cloud. “TPU v5p Supercomputing Details.” Google Cloud Technical Docs, 2026.
[7] Google Cloud. “Trillium (TPU v6e) Architecture & Matrix Engine Scaling.” Google Cloud Technical Docs, 2024.
[8] Google Cloud. “TPU7x (Ironwood) Architecture Specification & Dual-Chiplet Topology.” Google Cloud Technical Docs, 2025.
[9] Google Cloud. “Planning your TPU quota and cluster topology.” Google Cloud Technical Docs, 2026.
[10] StorageReview. “Google TPU 8t and TPU 8i Announced: Splitting Training and Inference Silicon.” StorageReview Enterprise Analysis, 2026.
[11] N. P. Jouppi, et al. “In-Datacenter Performance Analysis of a Tensor Processing Unit.” ACM/IEEE 44th Annual International Symposium on Computer Architecture (ISCA), 2017.
[12] Google DeepMind. “Optimizing Ragged Attention and KV-Cache Memory on Systolic Tensor Architectures.” Technical Whitepaper, 2025.

Read More Systems Research & Engineering

Link copied to clipboard!