Skip to content
AI Engineer Guide.

concept · concepts · intermediate · concept 33 of 33

Scaling laws

0 of 33 complete

Why it matters

Training runs cost real money and take weeks, so deciding how big a model to train and how much data to feed it cannot be done by trial and error at the top end. Scaling laws turn that decision into a fit: measure small runs, estimate the curve, and choose the allocation that minimises loss for the budget. They are the reason teams train a family of sizes rather than one model, and the reason data budgets grew faster than parameter counts.

Mental model

A scaling law is a recipe's yield table: it tells you how much output to expect from given amounts of each ingredient, fitted from small batches. The metaphor breaks because the table is only valid inside the range of batches you actually cooked, and a new ingredient — better data, a different architecture — changes the recipe rather than just the quantity.

How it works

The frontier bends toward a floorInteractive

The interactive figure loads with JavaScript. The text version below describes the same frame.

Figure. Loss along the compute-optimal frontier against training compute, on log10 compute and linear loss, using the fitted constants of Hoffmann et al. (2022). The solid segment covers the illustrative fit range; the dashed segment is the same law extrapolated. The shape is the claim, not any single loss value.

What to notice. Over the fitted three decades the curve drops 1.21 nats, 0.4 per decade on average: 0.55 in the first decade and 0.27 in the last. It bends toward the 1.69-nat floor rather than crossing it. Extrapolated five decades, the same law predicts 1.80 nats at 1e26 — a planning hypothesis, not a measurement. Every point is the compute-optimal allocation at that budget, so the promise is loss per unit of compute, not capability.

Text version

Loss from 1.6 to 3.8 nats against training compute from 1e18 to 1e26 on a log10 axis. The fitted solid segment falls from 3.53 nats at 1e18 to 2.33 at 1e21; the dashed extrapolation continues to 1.80 at 1e26, approaching a 1.69-nat floor marked by a horizontal line. A vertical line marks the end of the fit at 1e21 and a marker labels the 1e21 optimum at 1.8B parameters. Three steps reveal the fit, the extrapolation and the allocation; a panel reports 1.8B parameters, 91B tokens and a 2.33-nat loss at 1e21.

  1. Fix the architecture, tokenizer, optimiser and data distribution you intend to scale, since the law is conditional on all of them.
  2. Train several small models across a range of sizes and token budgets, holding other factors constant.
  3. Fit loss as a power law in parameters and data, including an irreducible term that represents the entropy floor of the corpus.
  4. Derive the compute-optimal allocation for your budget by minimising the fitted loss subject to the FLOPs constraint.
  5. Validate the prediction at one intermediate scale before committing the full run.
  6. Re-fit whenever the data mixture or architecture changes materially, and treat the old extrapolation as void.

Trade-offs

ChoiceGood forCostFails when
Compute-optimal allocationBest loss for a fixed training budgetSmaller models that are expensive to serve for high trafficInference volume dominates total cost
Oversized model, fewer tokensCheaper training, strong per-token capabilityUndertrained weights, poor loss per parameterYou have data but limited compute
Small model, many tokensCheap inference, strong data efficiencyLong training time and higher training costLatency budget allows a larger model
Extrapolating far beyond the fitPlanning without large pilot runsPrediction error grows quicklyArchitecture or data changed since the fit
Trade-off mapInteractive

Good for

Best loss for a fixed training budget

Cost

Smaller models that are expensive to serve for high traffic

Fails when

Inference volume dominates total cost

  1. Inference volume dominates total cost
  2. You have data but limited compute
  3. Latency budget allows a larger model
  4. Architecture or data changed since the fit

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Compute-optimal allocation is good for Best loss for a fixed training budget, costs Smaller models that are expensive to serve for high traffic, and fails when Inference volume dominates total cost. Oversized model, fewer tokens is good for Cheaper training, strong per-token capability, costs Undertrained weights, poor loss per parameter, and fails when You have data but limited compute. Small model, many tokens is good for Cheap inference, strong data efficiency, costs Long training time and higher training cost, and fails when Latency budget allows a larger model. Extrapolating far beyond the fit is good for Planning without large pilot runs, costs Prediction error grows quickly, and fails when Architecture or data changed since the fit

Failure modes

  • Law fitted across changed data. The extrapolation overpredicts because the corpus or tokenizer changed. Debug: re-run one small point under the new pipeline and compare against the predicted curve.
  • Compute counted without utilisation. The budget assumes peak FLOPs while the run achieves a fraction of them, so the chosen allocation is not the one trained. Debug: measure achieved FLOPs per step and recompute the effective budget.
  • Loss compared across tokenizers. Two fits look inconsistent because loss per token is not the same unit. Debug: convert to bits per character before comparing curves.

In production

Keep the fitted law and its validity range beside the training config, and re-validate at each scale step. Track achieved compute utilisation during the run, because a law that assumes peak hardware silently misprices the whole plan.

Common misconceptions

People think: Scaling laws say bigger is always better.

Actually: They say loss falls predictably with scale for a fixed budget; beyond the compute-optimal point, extra parameters without extra data buy very little.

People think: A scaling law predicts capability.

Actually: It predicts average loss on the training distribution. Specific abilities can appear abruptly or fail to appear, and the law says nothing about either.

Interview angle

A strong answer states the power-law form, the conditional nature of the fit, and the compute-optimal allocation result, then separates loss prediction from capability prediction. It should mention data quality and validation at scale. Weak answers say 'models improve with scale' without describing the allocation problem or the limits of extrapolation.

Sources