Tensor Processing Units: Architecture, Generations and Comparison with GPUs
“Hardware should be radically simplified around dense matrix arithmetic, while compiler software bears the burden of scheduling and synchronization.”
Google's Tensor Processing Unit (TPU) trades GPU-style hardware warp schedulers and cache hierarchies for massive $256 \times 256$ systolic arrays, software-managed scratchpad memory (VMEM), ahead-of-time XLA scheduling, and 9,216-chip optical circuit-switched 3D tori.
Executive Summary
A Tensor Processing Unit (TPU) is a custom application-specific integrated circuit (ASIC) designed by Google specifically for the matrix arithmetic at the foundation of modern deep neural network workloads. Google has deployed seven production generations since 2015 and announced an eighth generation that structurally bifurcates training and inference into distinct physical chips. This report analyzes how a TPU executes workloads, what microarchitectural changes emerged across major generations, how thousands of chips assemble into unified pods via optical circuit switches, and how the architecture fundamentally compares with contemporary GPUs.
Key Findings
- Design: A TPU dedicates the vast majority of its silicon real estate to a small number of large systolic-array matrix multiply units (MXUs), fed directly by software-managed on-chip SRAM scratchpad memory (VMEM) and scheduled deterministically ahead of time by the XLA compiler. It deliberately omits the hardware warp schedulers, dynamic dispatch units, and multi-level cache hierarchies found in GPUs [1], [10].
- Performance Trajectory: Peak BF16 throughput per chip grew from 123 TFLOPs on TPU v3 (2018) to 2,307 TFLOPs on Ironwood (2025)—an $18.8\times$ increase per chip, further reaching 4,614 TFLOPs in native FP8 [3], [8].
- Scale: An Ironwood pod interconnects up to 9,216 chips as a single coherent scale-up domain through a 3D torus constructed from $4 \times 4 \times 4$ cubes, with reconfigurable Optical Circuit Switches (OCS) joining the cubes dynamically [1], [8].
- Hardware Specialisation: The eighth generation (TPU 8t and 8i) separates training from inference, optimizing precision (native FP4 on 8t), memory hierarchies (384 MB on-chip SRAM on 8i for KV caching), and network topologies [10].
- Comparison with GPUs: The most decisive differences reside at system scale: a single TPU scale-up domain spans thousands of direct-routed chips rather than tens of GPUs behind InfiniBand/RoCE fabrics. Published figures often conflate sparse vs. dense numbers and unidirectional vs. bidirectional interconnect bandwidth [4].
Basis of the Figures
Unless explicitly noted otherwise, specifications derive from primary Google Cloud architecture documentation [1]–[9] and peer-reviewed IEEE/ACM ISCA publications. Technical details for TPU 8t and 8i originate from primary Google technical disclosures and analytical industry reporting [10]. All specifications reflect verified status as of November 2026.
1. How a TPU Works
1.1 Systolic Design Principle
The computational heart of a TPU is a systolic array: a physical 2D grid of multiply-accumulate (MAC) units hardwired directly to their orthogonal neighbors. In a traditional processor, intermediate matrix calculations must continually write back to registers or cache memory. In a systolic array, data streams continuously across the grid, handing intermediate partial products directly to adjacent accumulators. Matrix multiplication completes without intermediate memory traffic [1].
| Processor | Execution Pattern & Bottleneck Analysis |
|---|---|
| CPU | Loads operands from L1/L2 caches into register files, executes ALU instructions, and writes back for every operation. Arithmetic intensity is low; memory bus bandwidth caps throughput (the von Neumann bottleneck). |
| GPU | Thousands of lightweight SIMT ALUs work in parallel across Streaming Multiprocessors (SMs). Intermediate results cycle through register files or shared memory banks. Hardware warp schedulers hide memory latency through aggressive multi-threading. |
| TPU | Thousands of multiply-accumulators form a continuous physical systolic grid. Stationary weights are loaded into cell registers, activations stream through the array, and accumulated products pass downstream. No intermediate memory access occurs during the core dot-product calculation. |
1.2 Core Components
A production TPU chip integrates one or more TensorCores with high-bandwidth memory (HBM), vector/scalar processing units, inter-chip interconnect links, and specialized SparseCores. Table 2 details the principal functional units and Figure 1 illustrates how data moves across the hierarchy.
| Component | Microarchitectural Function & Dimensionality |
|---|---|
| TensorCore | The primary compute core containing the systolic Matrix Multiply Units (MXUs), a wide vector unit, a scalar control unit, and dedicated Vector Memory (VMEM) scratchpad. |
| MXU (Matrix Multiply Unit) | The systolic compute engine. Dimensioned at $128 \times 128$ up to TPU v5p (16,384 MACs) and scaled to $256 \times 256$ on v6e (Trillium) and TPU7x (Ironwood), quadrupling density to 65,536 MACs per unit. |
| Precision Engine | Inputs process as Bfloat16, with lower-precision native formats on modern silicon (INT8 on v5e/v6e; native FP8 on v5p/Ironwood; native FP4 on 8t). Accumulation is strictly maintained in FP32. |
| Vector & Scalar Units | Vector units compute elementwise activations (GELU, SwiGLU), LayerNorm/RMSNorm, and Softmax reductions. The scalar unit manages loop branching, instruction dispatch, and address generation. |
| HBM & DMA Subsystem | High-Bandwidth Memory stacks store global model weights, optimizer states, and KV-cache tensors. Data streams to on-chip VMEM scratchpads through asynchronous hardware DMA queues (infeed/outfeed). |
| SparseCore | A specialized asynchronous dataflow coprocessor engineered to accelerate sparse, memory-bandwidth-bound embedding table lookups in large recommendation models (DLRMs). v5p and TPU7x integrate four per chip; v6e integrates two. |
1.3 The Role of the XLA Compiler
A TPU offloads control-flow complexity from silicon to software. Unlike GPUs, which expend significant die area on hardware warp schedulers, dynamic scoreboard trackers, and out-of-order execution logic, a TPU operates deterministically under the ahead-of-time (AOT) control of Google's XLA (Accelerated Linear Algebra) compiler [10].
XLA ingests computational graphs from high-level frameworks (JAX, PyTorch via PyTorch-XLA), decomposes operations into primitives, performs aggressive operator fusion, and directly schedules every cycle of computation, DMA transport, and inter-chip collective communication. Because execution timing is completely static and cycle-accurate, thousands of chips remain lockstep-synchronized during distributed all-reduce operations without incurring runtime communication jitter.
The primary trade-off of a massive fixed-dimension systolic array is matrix dimension mismatch. When operational matrix dimensions are not integer multiples of the systolic grid ($128$ or $256$), XLA must pad dimensions with zeros. Unpadded irregular token counts or dynamic sequence lengths cause execution lanes to sit idle. Addressing this limitation motivated ragged paged-attention algorithms engineered specifically to pack irregular LLM decode batches into dense, tile-aligned systolic blocks [12].
2. How the Major Generations Differ
Google's architectural progression reflects an initial focus on internal recommendation and vision models, transitioning to massive distributed LLM pre-training, and culminating in specialized dual-silicon systems.
| Generation | Year | Compute Engine | Peak Compute (TFLOPs) | HBM Capacity & BW | Interconnect | Max Pod Size |
|---|---|---|---|---|---|---|
| TPU 8i [10] | 2026* | 2 TensorCores; Collectives Engine | Serving Optimized | 288 GB HBM + 384 MB SRAM | Boardfly Mesh | 1,152 chips |
| TPU 8t [10] | 2026* | 1 TensorCore; SparseCores; FP4 | ~4,600+ BF16 / FP4 | 216 GB HBM | 3D Torus (19.2 Tb/s) | 9,600 chips |
| Ironwood (TPU7x) [8] | 2025 | Dual Chiplet (2 TC, 4 SparseCores) | 2,307 BF16 / 4,614 FP8 | 192 GiB (7.37 TB/s) | 3D Torus (1,200 GB/s) | 9,216 chips |
| v6e (Trillium) [7] | 2024 | 1 TC (two $256 \times 256$ MXUs), 2 SC | 918 BF16 / 1,836 INT8 | 32 GB (1.6 TB/s) | 2D Torus | 256 chips |
| v5p [6] | 2023 | 2 TC (four $128 \times 128$ MXUs), 4 SC | 459 BF16 / 918 FP8 | 95 GiB (2.76 TB/s) | 3D Torus with OCS | 8,960 chips |
| v5e [5] | 2023 | 1 TC (four $128 \times 128$ MXUs) | 197 BF16 / 393 INT8 | 16 GB (819 GB/s) | 2D Torus | 256 chips |
| v4 [4] | 2021 | 2 TC (four $128 \times 128$ MXUs), 4 SC | 275 BF16 | 32 GiB (1.2 TB/s) | 3D Torus with OCS | 4,096 chips |
| v3 [3] | 2018 | 2 TC (two $128 \times 128$ MXUs each) | 123 BF16 | 32 GiB (900 GB/s) | 2D Torus (Copper) | 1,024 chips |
2.1 What Changed at Each Step
Deployed strictly for inferencing DeepMind and Google Search workloads. Featured a single 65,536-MAC 8-bit matrix unit operating at 92 TOPS with 28 MiB of software-managed SRAM, demonstrating that deterministic execution suits tail-latency bounds far better than caches and out-of-order pipelines [11].
Added support for backpropagation, co-invented the Brain Floating Point (bfloat16) format, integrated High-Bandwidth Memory (HBM), and linked pods over a direct-connected 2D torus scaling to 1,024 liquid-cooled v3 chips (123 BF16 TFLOPs per chip) [2], [3].
Replaced rigid copper cables at cluster boundaries with dynamic optical circuit switches. Each chip integrated 6 Inter-Chip Interconnect (ICI) links, forming reconfigurable $4 \times 4 \times 4$ cubes scaling to 4,096-chip 3D tori. Introduced specialized SparseCores for recommendation embeddings [4].
Split development into two parallel silicon tiers: v5e (cost-optimized serving and fine-tuning with 1 TensorCore and 2D torus in 256-chip slices) and v5p (flagship training supercomputer with 2 TensorCores, 95 GiB HBM, and 8,960-chip 3D torus scaling) [5], [6].
Quadrupled MAC density per MXU by expanding systolic arrays from $128 \times 128$ to $256 \times 256$, lifting peak BF16 compute per chip from 197 to 918 TFLOPs ($4.7\times$ the v5e). Retained high-density 256-chip 2D torus networking for serving and fine-tuning [7].
Google's first chiplet-based TPU with native FP8. Each physical socket packages two chiplets (each with its own TensorCore, two SparseCores, and 96 GiB of HBM), exposed to frameworks such as JAX as two runtime devices. Delivers 2,307 BF16 TFLOPs (4,614 FP8 TFLOPs) and 192 GiB HBM at 7.37 TB/s across 9,216-chip 3D tori at $2\times$ the performance per watt of Trillium [8], [9].
The 8t focuses exclusively on massive pre-training with native FP4 arithmetic, 216 GB HBM, and inter-chip bandwidth doubled to 19.2 Tb/s per chip. The 8i re-engineers inference serving: it replaces SparseCores with a hardware Collectives Acceleration Engine, integrates 384 MB of on-chip SRAM to hold active KV-caches alongside 288 GB HBM, and adopts a Boardfly topology reducing worst-case hops across 1,024 chips from 16 to 7 [10].
3. Scaling: Pods, Torus Links and Optical Switching
A TPU supercomputer is fundamentally architected as a single cohesive fabric: thousands of chips connect directly to their neighbors via low-latency Inter-Chip Interconnect (ICI) links without staging packets through intermediate InfiniBand adapters or Ethernet switches. On Ironwood, each chip provides 6 bidirectional links running at 200 GB/s per link, yielding 1,200 GB/s of direct interconnect bandwidth per chip [8].
3.1 Cubes and Optical Circuit Switching (OCS)
Physical packaging groups 64 chips into a $4 \times 4 \times 4$ cube wired internally via passive copper cables. Cube boundaries are joined through optical transceivers and reconfigurable Optical Circuit Switches (OCS) into a 3D torus or twisted torus on v4, v5p, and Ironwood [1].
An OCS uses micro-electro-mechanical system (MEMS) mirrors to redirect light beams directly between optical fibers without optical-electrical-optical conversion. This delivers three fundamental architectural capabilities:
- Dynamic Topology Shaping: Slices of different shapes can be carved dynamically from a pod into arbitrary $X \times Y \times Z$ torus dimensions tailored to specific 3D parallelism configurations (tensor, pipeline, and data parallel axes).
- ICI Fault Resiliency: When a chip, transceiver, or optical link fails, OCS mirrors re-route traffic around the defective link or switch in seconds, restoring cluster execution without physical technician dispatch [4].
- Twisted Torus Bisection: Re-pointing the wrap-around links transforms an asymmetric 3D torus into a symmetric twisted torus, providing roughly 70% more theoretical bisection bandwidth for a twisted $4 \times 4 \times 8$ slice [1].
3.2 Slices and Multislice
In Google Cloud terminology, a slice designates an allocation of chips within a single pod interconnected entirely through ICI. Because ICI operates at optical line rates with sub-microsecond latency, operations inside a slice execute as a single unified hardware supercomputer.
Multislice extends scaling beyond physical pod boundaries. While ICI handles latency-critical tensor-parallel and pipeline-parallel communication inside each slice, data-parallel gradient synchronizations or mega-batch reductions bridge slices asynchronously over Google's Jupiter Data Center Network (DCN). On v5p, a single slice spans 6,144 nodes, while Multislice coordinates up to 18,432 chips [6].
3.3 Topology and the Eighth Generation
The announced TPU 8t and 8i diverge in network topology to optimize for diametrically opposed traffic patterns:
- TPU 8t (Training): Preserves the 3D Torus architecture. Training is dominated by regular, high-volume all-reduce primitives along dimensional rings. Doubling ICI bandwidth to 19.2 Tb/s per socket allows the 8t to sustain near-linear scaling up to 9,600 chips [10].
- TPU 8i (Inference Serving): Replaces the 3D torus with a Boardfly topology (a variant of the hierarchical Dragonfly topology). Large language model serving requires rapid cross-node tensor-parallel reductions with minimum latency rather than maximum ring volume. Boardfly guarantees that the worst-case network diameter across a 1,024-chip serving cluster drops from 16 hops (in a 3D torus) to only 7 hops [10].
4. TPU and GPU Compared
While both GPUs and TPUs process modern tensor workloads, they embody fundamentally contrasting systems engineering philosophies:
| Dimension | Google TPU (Ironwood / 8-Series) | NVIDIA GPU (Hopper H100 / Blackwell B200) |
|---|---|---|
| Compute Core | Few large 2D systolic arrays ($256 \times 256$ MACs) + wide SIMD vector unit. | Many Streaming Multiprocessors (SMs), each with 64/128 CUDA cores + 4 Tensor Cores. |
| Scheduling & Dispatch | Static, ahead-of-time VLIW instruction scheduling by XLA compiler. No warp schedulers. | Dynamic runtime hardware warp schedulers (32 threads per warp) hiding memory stalls. |
| Memory Hierarchy | Flat HBM + large software-managed scratchpad SRAM (VMEM). Zero hardware L1/L2 tags. | Multi-tiered hardware cache hierarchy (L1/Shared Memory, massive L2 cache, HBM). |
| Scale-Up Domain | Direct-routed 3D Torus via ICI & OCS across 9,216 chips per pod. | NVLink/NVSwitch domain typically spans 8 to 72 GPUs; wider scale requires InfiniBand/RoCE. |
| Network Switching | Optical Circuit Switches (OCS) switching pure photons without transceivers. | Electronic packet switches (NVSwitch, Quantum-2 InfiniBand, Spectrum-X Ethernet). |
| Execution Model | Deterministic, cycle-accurate timing. Highly predictable collective latency. | Asynchronous, warp-divergent execution. Highly adaptable to dynamic kernel execution. |
| Precision Roadmap | Pioneered Bfloat16; native FP8 on Ironwood; native FP4 on TPU 8t. | Pioneered TensorFloat-32 (TF32); 4th-Gen FP8 (Hopper); 2nd-Gen NV-FP4 (Blackwell). |
4.1 Reading Published Comparisons
Published benchmark head-to-heads between TPUs and GPUs frequently introduce confounding variables. Engineers evaluating published claims must normalize three common discrepancies:
- Sparse vs. Dense FLOPS: NVIDIA datasheets frequently headline Tensor Core throughput assuming 2:4 structured sparsity, which mathematically doubles theoretical peak FLOPS by assuming 50% zero-weights. TPU specifications quote strictly dense matrix throughput. A 2,307 TFLOP Ironwood chip should be compared to the dense FP16/BF16 rating of an H100 (989 TFLOPs), not its 1,979 TFLOP sparse figure.
- Unidirectional vs. Bidirectional Interconnect: Google documentation specifies ICI bandwidth in aggregate bidirectional terms (e.g., 1,200 GB/s on Ironwood across 6 links, meaning 100 GB/s transmit + 100 GB/s receive per link direction). GPU specifications alternate between per-link bidirectional and unidirectional NVLink numbers.
- Cluster Power & PUE: Because TPUs eliminate electronic InfiniBand switches inside the pod and replace them with passive optical circuit switches, pod-level cooling and power overhead (PUE) are significantly lower than equivalent multi-switch GPU superclusters [4].
5. Choosing a Platform and a First Program
5.1 Workload Fit
Neither architecture is universally superior. Selection depends on mathematical workload structure, software ecosystem, and cluster scale:
- When TPU Excels: Large dense transformer training (LLMs, multimodal backbones) where sequence shapes are stable; ultra-large distributed jobs requiring synchronous all-reduce across thousands of nodes without network congestion; teams natively developing in JAX or PyTorch-XLA; workloads where deterministic cost-per-token dominates.
- When GPU Excels: Inference serving with extreme dynamic batching and ragged sequence lengths; models relying heavily on custom hand-written CUDA/Triton kernels (e.g., non-standard attention variants, state-space models like Mamba); graph neural networks and irregular sparse architectures; teams requiring immediate local desktop or on-premise prototyping.
5.2 A First Program in JAX
JAX is the native programming model for the TPU. The following program demonstrates tensor device enumeration, ahead-of-time compilation via XLA, and multi-device 2D matrix sharding across a TPU mesh:
import jax
import jax.numpy as jnp
from jax.sharding import Mesh, PartitionSpec as P, NamedSharding
# 1. Enumerate available physical TPU devices
devices = jax.devices()
print(f"Connected TPU Devices: {devices}")
print(f"Device Kind: {devices[0].device_kind}")
# 2. Form a 2D logical device mesh (e.g. data & model parallelism)
# Assuming 8 devices: 2 data-parallel axes, 4 model-parallel axes
mesh = Mesh(devices=jax.devices().reshape(2, 4), axis_names=('data', 'model'))
# 3. Define 2D sharded matrices
# Matrix X is sharded along data parallel axis; W is sharded along model axis
dim_b, dim_k, dim_out = 8192, 4096, 8192
sharding_x = NamedSharding(mesh, P('data', None))
sharding_w = NamedSharding(mesh, P(None, 'model'))
x = jax.random.normal(jax.random.PRNGKey(0), (dim_b, dim_k), dtype=jnp.bfloat16)
w = jax.random.normal(jax.random.PRNGKey(1), (dim_k, dim_out), dtype=jnp.bfloat16)
# Transfer tensors to TPU HBM with explicit sharding layouts
x_sharded = jax.device_put(x, sharding_x)
w_sharded = jax.device_put(w, sharding_w)
# 4. Ahead-Of-Time (AOT) Compiled Matrix Multiplication via XLA
@jax.jit
def parallel_matmul(a, b):
# Systolic array executes dot-product; outputs sharded along ('data', 'model')
return jnp.matmul(a, b)
# Execute on physical silicon
out = parallel_matmul(x_sharded, w_sharded)
out.block_until_ready()
print(f"Output Matrix Shape: {out.shape}, Sharding: {out.sharding}")
6. Conclusion
Google's Tensor Processing Unit represents an architectural thesis: hardware should be radically simplified around dense matrix arithmetic, while compiler software should bear the burden of scheduling and synchronization. By substituting complex hardware warp managers and cache hierarchies with massive systolic arrays, software-managed scratchpads, and direct optical circuit switching, the TPU achieves extraordinary performance density and scales thousands of nodes into unified, direct-routed supercomputers.
As the industry bifurcates into distinct pre-training scaling limits (demanding 10,000-node FP4 fabrics) and high-concurrency low-latency serving (demanding massive SRAM KV-caches and Boardfly topologies), the eighth-generation TPUs solidify this specialized, domain-specific trajectory.
TPU deployments are constrained to Google Cloud and internal Google datacenters; on-premises hardware ownership is not commercially supported. Workloads that cannot be framed as dense tensor operations or that require highly dynamic control flow will encounter lower hardware utilization due to systolic padding constraints.