AI Accelerators Explained: GPUs, TPUs, NPUs and More

Training and inference

Training is the process of adjusting a model's weights, the numbers that define it, until it performs a task well. It runs the model forward, measures the error, and then runs backward to update the weights. It generally needs more numerical precision than inference, large memory for intermediate results, and many chips linked by fast connections so they can work as one machine.

Inference is using the trained model to produce answers. It runs forward only. The goals are low delay for each request, high total throughput and low cost per result. Inference happens every time anyone uses an AI service, so for popular models it eventually accounts for most of the compute.

Inference for language models has two phases. Prefill reads the prompt and is limited mainly by arithmetic. Decode produces the answer one token at a time and is limited mainly by memory bandwidth. Chips are designed with different balances between these.

GPUs

GPUs are the default AI accelerator. They contain thousands of arithmetic units and dedicated matrix units, called tensor cores in NVIDIA's chips, and they use high bandwidth memory.

Their main strength is flexibility. The same GPU can train and serve many types of model and run other scientific work. Their other strength is software. NVIDIA's CUDA has the largest ecosystem, and AMD's ROCm is its main competitor.

NVIDIA is the market leader, with AMD as the main alternative. Their data centre parts are expensive and in high demand.

TPUs and other custom accelerators

A TPU is Google's tensor processing unit, a custom chip designed around matrix multiplication using systolic arrays. Thousands of TPUs are linked into large groups called pods with a custom network. Recent generations have used optical circuit switches to reconfigure the links.

Large cloud providers build their own chips for similar reasons. Amazon has Trainium and Inferentia, Microsoft has Maia, and Meta has MTIA. A custom chip tuned to the provider's own workloads can reduce cost and lessen reliance on one supplier.

NPUs

NPU stands for neural processing unit. It usually means a small AI block inside a phone, laptop or other device chip. Examples include Apple's Neural Engine and the NPUs in Qualcomm, Intel and AMD processors.

NPUs target low power. They run tasks such as photo enhancement, speech recognition and small language models on the device itself, without sending data to a server. Microsoft requires an NPU rated at 40 TOPS or more for its Copilot+ PC label. TOPS means trillions of operations per second, a peak figure that real workloads rarely reach.

Inference ASICs

An ASIC, or application-specific integrated circuit, is built for one purpose. Several start-ups build ASICs just for inference. The idea is that fixing the design around one task allows better speed or energy use.

Groq built a chip that relies on large on-chip SRAM and a compiler that schedules every step in advance, which gives predictable delay. Nvidia licensed Groq's technology in late 2025, and Groq now focuses more on running inference as a service. Etched has designed a chip specifically for transformer models, the architecture behind most large language models. d-Matrix combines memory and compute. Tenstorrent, SambaNova and others take their own routes.

The risk is obsolescence. If model designs change, a chip hard-wired for today's design may lose its advantage. General chips can adapt, and specialised ones cannot adapt as easily.

Wafer-scale chips

A silicon wafer is the round slice on which many chips are made, normally cut up afterwards. Cerebras instead keeps a whole wafer as one processor. Its third-generation Wafer Scale Engine has about four trillion transistors, 900,000 cores and 44 gigabytes of on-chip memory.

Keeping everything on one piece of silicon avoids slow links between chips and gives very high memory bandwidth. Making it work needs special solutions for power, cooling and defects, since some of the silicon in any wafer is faulty. Cerebras handles this by routing around bad cores.

How to choose

The right accelerator depends on the job. For research and varied workloads, GPUs are the safe choice. For stable, large-scale workloads on one cloud, a custom chip may cost less. For devices and sensors, an NPU or edge accelerator fits the power budget. For very low delay inference, specialised chips are worth testing.

Compare chips on delivered results for your own model, not on peak numbers. Useful measures are tokens per second, cost per million tokens, and energy per result. The MLPerf benchmarks from MLCommons offer standard tests, though they cover only some workloads.

Finally, count the cost of software. Moving a model from one platform to another takes engineering time, and that cost often outweighs a modest hardware saving.