TPU (Tensor Processing Unit) — AI Glossary

Reference details

Related terms

• NVIDIA H100
• GPU Memory (VRAM)
• Distributed Inference

How it works

1. The chip is built around a systolic array for matrix multiplication: data flows through a fixed grid of multiply-accumulate units so that each value loaded is reused across many operations rather than being fetched per operation.
2. That design trades generality for efficiency on one shape of work. Large, regular, dense matrix operations map onto it extremely well, and irregular or control-heavy computation does not map onto it at all.
3. Programs are compiled ahead of time into a static graph, so shapes must be known at compile time and dynamic shapes force recompilation — which is why workloads with variable sequence lengths need bucketing.
4. Chips are connected by a dedicated high-bandwidth interconnect into pods, and that interconnect rather than the per-chip arithmetic is what makes large-scale synchronous training practical.
5. On-chip memory per accelerator is the hard constraint for serving: a model must be partitioned across chips once its weights and cache exceed one chip's capacity, and the partitioning is a compiler-level decision.
6. The programming model is reached through frameworks that compile to it rather than through hand-written kernels, which lowers the barrier for standard architectures and raises it for unusual ones.

Example: Why per-chip memory decides the partitioning

Take the 95GB of high-bandwidth memory per chip stated for the current generation. A 70B model in 16-bit weights occupies 70e9 x 2 = 140e9 bytes, 140GB — more than one chip holds, so it must be split across at least 140 / 95 = 1.47, meaning 2 chips, and every forward pass then crosses the interconnect. The same model quantised to 8 bits is 70e9 x 1 = 70GB, which fits one chip with 95 - 70 = 25GB left for the key-value cache and activations. That single arithmetic step changes the deployment shape completely: one chip with no cross-chip communication against two chips with a collective on every layer. And the 25GB of headroom is itself the next constraint, since cache grows with context and concurrency — at 0.5MB per token for a model of this class, 25GB supports 25e9 / 524,288 ≈ 47,700 tokens of cache in total, which is one long context or several short ones but not both. The pattern generalises: on this hardware the first question is always how the model divides into the per-chip memory, because that determines whether you are running a single-chip workload or a distributed one.

Commonly confused with

This is an ASIC for dense matrix work, not a GPU with a different label, and the differences are the ones that bite. A GPU is a general parallel processor with a mature kernel ecosystem, dynamic shape support and broad framework coverage; this is a fixed-function accelerator reached through a compiler, where static shapes are required and an unsupported operation is a blocker rather than a slow path. So porting is not a configuration change: a model built on standard layers compiles and runs well, while one using custom kernels or data-dependent control flow may not compile at all.

When to use it

Reach for it when:

• Large-scale training of standard architectures, where the systolic design and the pod interconnect are the point.
• Workloads already in a framework that compiles to it, where the port is genuinely a backend change.
• Static, regular shapes — fixed sequence lengths or a small bucket set — which the ahead-of-time compilation requires.
• Serving models that fit the per-chip memory after quantisation, which avoids cross-chip communication entirely.

Reach for something else when:

• Models with custom kernels or data-dependent control flow, which may not compile.
• Highly variable shapes without bucketing, where recompilation dominates.
• Small or experimental workloads, where the compilation and tooling overhead exceeds the benefit.
• Assuming a GPU port is a configuration change, since the compiler's constraints are architectural rather than tunable.

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to TPU (Tensor Processing Unit) — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path