AI Accelerators

Explainer

AI Accelerators explained

An AI accelerator is a chip, or a part of a chip, designed to run machine learning faster or with less energy than a general-purpose processor. The category includes GPUs, TPUs, neural processing units (NPUs) and chips from many startups.

Most accelerators speed up the same thing: multiplying large arrays of numbers. They do this with many parallel multiply units, lower-precision number formats such as INT8 or FP8, and fast on-chip memory. Some target training, which builds a model, and others target inference, which runs it.

How it differs from the usual approach

A CPU is built for flexibility and a GPU for broad parallel work. An AI accelerator narrows the design to neural networks, trading generality for efficiency. Startups try different ideas, such as very large chips, on-chip memory in place of HBM, or dataflow designs. Each choice shifts performance on some models and not on others, and the software stack matters as much as the chip.

Where it is used

• Training large models in cloud data centers.
• Serving chatbots and image generators at low cost per query.
• NPUs in phones and laptops for on-device features.
• Cars, cameras and robots that run models locally.
• Enterprise servers running private or regulated AI workloads.

What to watch

• Inference, which is repeated for every user query, is a growing share of AI computing, and many new chips target it.
• Cerebras builds a wafer-scale chip; its WSE-3 has about 900,000 cores and four trillion transistors on one wafer.
• Cloud providers are building their own accelerators alongside buying GPUs.
• Software maturity decides adoption. A fast chip that is hard to program often loses to a slower one that works with existing tools.

Frequently asked questions