Scaling Laws

The empirical bedrock of the AI era is a set of curves: train transformers across orders of magnitude of compute, data, and parameters, and prediction loss falls along strikingly clean power laws. The discovery transformed the economics of the field — capability stopped being a research lottery and became a forecastable function of investment. When labs raise billions for compute, these curves are the underwriting.

The laws also dictate proportions. DeepMind's Chinchilla result showed that early large models were badly under-trained on data — for a fixed compute budget, loss minimizes near roughly twenty tokens of training data per parameter, meaning smaller models trained far longer beat giant models trained briefly. The finding redirected industry strategy overnight and explains today's pattern of compact models with enormous training runs, which also inference cheaper forever after.

Scaling laws predict aggregate loss smoothly, but individual capabilities can arrive in jumps (emergence) — so the curves forecast the trendline while specific abilities still surprise. Two further caveats discipline the optimism: high-quality training data is a finite resource whose limits increasingly bind frontier runs, and a falling loss curve is not the same as proportional gains on the tasks your business cares about. The law guarantees better prediction, not better economics.

The newest chapter scales a different variable: inference-time compute. Reasoning models that think longer on harder problems exhibit their own scaling behavior — more deliberation, better answers — opening a second capability axis that doesn't require retraining anything. Strategy now navigates both curves: training scale (capital-intensive, lab-controlled) and test-time scale (operationally controlled, billed per query) — with right-sizing across them the core cost-capability decision of deployment.