Here’s the secret the spec sheets bury: a modern GPU can do math far faster than anything can deliver the numbers to work on. So the chips spend a lot of their life waiting. Understand that, and you understand most of what AI hardware engineers actually fight about.
In Part 1 we met the GPU — thousands of cores doing math in parallel. The problem is that those cores are hungry. They burn through numbers faster than memory can hand them over. Engineers call this the memory wall: the chip isn’t the limit, the plumbing is. A lot of expensive silicon sits idle simply waiting for data to show up.
Picture a chef who can chop ingredients in a blur, but the ingredients arrive one at a time on a slow conveyor belt. The chef’s speed barely matters — the belt sets the pace. In AI, the GPU is the chef and memory bandwidth is the belt.
Data inside an AI machine travels over several different "roads," and they run at wildly different speeds. The closer the memory is to the GPU, the faster the road. Tap each bar to see what it is and why it matters.
GPU ↔ HBM memory: The memory bolted right onto the GPU. By far the fastest path — and still the thing the cores wait on.
Bars use a log scale — each step is many times faster than the last. Figures are rough, current-generation numbers.
A single GPU can’t hold the biggest AI models — they’re too large. So you split a model across dozens or even thousands of GPUs. But that only works if the chips can talk to each other almost as fast as they think. The normal cable between chips (PCIe) is too slow for that. NVIDIA’s answer is NVLink: a private, super-fast link that stitches many GPUs into what behaves like one enormous chip.
This is a big part of why NVIDIA is so hard to displace. A rival might build a GPU that matches NVIDIA’s on paper, but if it can’t link thousands of them together efficiently, it can’t run the largest models. The network between chips has become as important as the chips themselves — which is why companies like Broadcom and Arista Networks matter to the AI story too.
The unit of AI compute is no longer "a chip." It’s a rack of chips wired together so tightly they act as one. The wiring — NVLink inside a server, high-speed switches between servers — is now a competitive moat in its own right.
Once you accept that data movement is the bottleneck, a lot of AI hardware design makes sense. Stack memory right on top of the GPU (that’s HBM, from Part 1). Keep data on the GPU instead of shuffling it back to the CPU. Wire chips together with the fastest link you can build. Almost every headline advance — bigger HBM, faster NVLink, smarter chip packaging — is really about one thing: keeping the cores fed.
Part 2 of the AI Compute 101 series. Next: From Sand to Superchip — how one of these chips is actually built.
For informational purposes only. Not investment advice. Bandwidth figures are approximate, current-generation values.
The memory wall is the point where a processor can do arithmetic faster than the memory system can supply operands. The chip is no longer the limit — the plumbing is. In AI servers this means GPU cores spend a large share of their time waiting for data rather than computing.
FLOPS measures peak arithmetic the chip could do; bandwidth determines how much data you can actually get to those cores. If the data cannot arrive fast enough, peak FLOPS is theoretical. Real throughput on large models is usually bandwidth-bound, which is why HBM generation matters more than headline compute.
NVLink is a high-speed interconnect that wires GPUs directly to each other instead of routing traffic through slower system buses. Large models do not fit in one GPU’s memory, so the GPUs must constantly exchange data. NVLink exists to stop that exchange from becoming the bottleneck.