All those chips do two completely different jobs, and almost every argument about AI’s costs and profits comes down to which one you’re talking about. One is teaching the model. The other is using it. They need different hardware, cost different amounts, and matter to very different parts of the economy.
Think of an AI model like a student. Training is the years of schooling — slow, brutally expensive, done once, and you never see it. Inference is the student answering a question on the job — fast, cheap per question, and happening constantly for the rest of its life. Same brain, two totally different activities. Toggle between them below to see how differently they stress the hardware.
Showing the model trillions of examples until it learns. Done once (then occasionally repeated). Enormously expensive.
Toggle between the two jobs. Same model — completely different demands on the hardware.
You’ll see AI compute measured in FLOPS — floating-point operations per second, basically "how many math sums per second." Training a frontier model can take more sums than a powerful computer could do in thousands of years if it ran them one at a time. That’s why training uses thousands of the most expensive GPUs, lashed together with the fast links from Part 2, running flat-out for weeks.
Training is a one-time megaproject. Inference is a forever-bill. A model might cost $100M to train once — then cost even more than that every year just to answer everyone’s questions. As AI products get popular, the spending quietly shifts from training to inference.
Here’s why investors care. Training rewards raw, top-end power — and that has been NVIDIA’s stronghold, because only its chips link together well enough to train the biggest models. But inference is a different game: it rewards being cheap and efficient at massive scale. That opens the door to rivals — AMD, plus custom chips that Google, Amazon, and Meta design themselves (often with Broadcom) to run their own models more cheaply.
So when you hear debates about whether the AI build-out is overdone, a lot of it is really this question: how much of future spending is training (winner-takes-most, NVIDIA-heavy) versus inference (more competitive, more spread out)? The honest answer is the mix is shifting toward inference over time — which is exactly why every big tech company is now designing its own chips.
A simple lens for any AI headline: is this about building the models (training) or running them (inference)? The first is a spike of upfront spending; the second is a meter that never stops running.
That’s the series. You now know the four things that explain almost every AI hardware story: the three parts (CPU, GPU, HBM), the memory wall, how a chip is built, and the two jobs it does. Next time you read an AI headline, you’ll know which piece it’s really about.
Part 4 of 4, AI Compute 101 series.
For informational purposes only. Not investment advice. Cost figures are rough, illustrative ranges.
Training is the process of teaching a model by showing it trillions of examples — a one-time megaproject costing tens or hundreds of millions of dollars. Inference is using the finished model to answer a query — individually cheap, but running continuously forever. They need different hardware and have opposite cost profiles.
Training is a large one-time cost; inference is a recurring one. A model that costs $100M to train can cost more than that annually just to answer queries once it is popular. Over a successful model’s life, inference dominates total spend.
FLOPS means floating-point operations per second — how many arithmetic sums a chip does per second. Training a frontier model takes more total operations than a powerful computer could complete in thousands of years running them one at a time, which is why training uses thousands of chips in parallel.