To train a single frontier AI model, tech giants can now burn through an estimated 10²⁶ FLOPs or more. That is more floating-point operations than the estimated number of grains of sand on all of Earth’s beaches.12
First off, what is a FLOP? It stands for floating-point operation: an arithmetic calculation, such as an addition or multiplication, on floating-point numbers. These formats represent values across a wide range, including fractions.
Why does this matter for LLMs? Their learned parameters, mainly weights, are numerical values. Training or running a model involves combining them with activations—the intermediate values produced as an input passes through the network—in large matrix operations. Much of this arithmetic uses floating-point formats.
Here, I’ll use FLOPs, with a lowercase s, for the number of operations. If you are following a recipe, how many steps does it take? FLOPS, or more explicitly FLOP/s, measures a rate: how many operations a GPU or TPU can perform per second. An advertised peak applies at a particular numerical precision, under specified conditions.3 The distinction matters: a training total tells us how much arithmetic was involved; FLOP/s tells us how quickly hardware can, in principle, perform it.
For dense-transformer training, a common estimate is C ≈ 6ND, where N is the number of parameters and D is the number of training tokens.4 Epoch AI puts Gemini 1.0 Ultra’s training compute at roughly 5 × 10²⁵ FLOPs and GPT-4’s at 2 × 10²⁵, labeling both estimates speculative. Both numbers give us a sense of the scale of training. They do not directly measure a model’s capability, tell us how efficiently the hardware performed that work, or predict how quickly the model will respond.5
A headline about 100,000 GPUs gives us a device count. Adding up their advertised FLOP/s gives a theoretical peak at a specified precision, not the rate a model will necessarily sustain. For training, model FLOPs utilization (MFU) compares the model’s estimated forward-and-backward operations per second with that aggregate peak, excluding extra work such as recomputing activations. PaLM reported 46.2% MFU on 6,144 TPU v4 chips. The remainder was not simply idle hardware; one percentage cannot tell us what is limiting performance.6
Similarly, 95% GPU utilization in nvidia-smi means a kernel was running during 95% of the sampled time. It does not mean 95% of peak arithmetic throughput.7
So what are the chips waiting for? In distributed training, devices exchange gradients and intermediate results. A slow device or network link can make dependent workers wait. Meta’s Llama 3 paper describes how a single straggler can slow thousands of GPUs.8
Within each accelerator, High Bandwidth Memory (HBM), a type of DRAM, supplies weights and activations to the compute cores. If the workload needs data faster than memory can deliver it, arithmetic capacity goes unused. That is a bandwidth problem; memory capacity is how much data fits. Some workloads also lack enough independent work to occupy all the arithmetic units. The limiting resource depends on the workload.3
Training efficiency does not automatically predict serving speed either. Processing a prompt offers parallel work, while standard autoregressive generation has dependencies between successive tokens. Batching requests can expose more work and improve data reuse, but must fit memory and latency constraints. What “faster” means matters: serving more requests and reducing the wait for an individual request are different goals.9
FlashAttention is one example of what software can change. It computes attention in tiles using fast on-chip SRAM, reducing HBM traffic and avoiding storing the full attention matrix there. It showed that reorganizing the computation could improve speed on the same hardware. The paper is worth reading for how it approaches this problem.10
That brings me to another question: if GPUs already offer so much compute, why are companies designing their own chips? OpenAI’s Jalapeño is a custom inference chip developed alongside its models and serving software. OpenAI describes this as a way to improve latency, energy efficiency and cost together, while continuing to use other companies’ accelerators.11
Modern GPUs already have specialized hardware for matrix operations.3 Custom silicon allows further choices around a workload: how data is stored and reused, which numerical formats to support, and how chips communicate. Google’s TPUs illustrate these choices through their matrix units, memory organization and inter-chip connections.12 Matrix multiplication dominates much of an LLM’s arithmetic, but activations, normalization and other operations also matter. Custom chips can improve arithmetic too. The point is to choose the balance of compute, memory and communication that suits the work.
Those choices come with development costs, software investment and specialization. Custom hardware makes sense when the gains at the intended scale justify them. For another workload, improving the software on existing hardware may be the better choice. More FLOPS help when arithmetic is the bottleneck. Before buying more silicon, I would want to know what the workload is actually waiting for.
I find the work around data movement as interesting as the chips themselves. I want to keep writing about how LLMs process data, the operational challenges that emerge at scale, and how different models change what we need from the systems running them. There is plenty here to come back to.