Definition
Scaling laws is Scaling laws are empirical rules showing that an AI model's error falls in a smooth, predictable way as you increase its size, training data and compute, which lets labs forecast results before training.
Scaling laws are the reason labs can plan a huge training run with some confidence. They describe how a model's performance improves as you give it more of three ingredients: parameters, training data and compute.
In 2020, researchers at OpenAI (Kaplan and colleagues) trained hundreds of language models of different sizes and found that the loss, a measure of prediction error, followed smooth power-law curves. More parameters, more data or more compute each helped, and the gains were predictable across many orders of magnitude.
Because the curves are smooth, a lab can train small models cheaply, fit a line through the results and estimate how a model a thousand times bigger would perform.
In 2022, a DeepMind team (Hoffmann and colleagues) re-ran the experiments and reached a different balance. Their model Chinchilla, at 70 billion parameters, outperformed the much larger 280 billion parameter Gopher trained with the same compute. Their conclusion was that many models were too big for the amount of data they saw, and that parameters and training tokens should grow together. A commonly cited rule of thumb from that work is about 20 training tokens per parameter. Researchers have since studied why the two papers disagreed, and part of the answer appears to be how carefully each tuned its training settings.
The Chinchilla ratio targets the cheapest way to reach a given quality in training. It ignores what happens afterward. If a model will serve billions of requests, it can be worth training a smaller model on far more data than the rule suggests, because every later inference call is cheaper. Many modern small models are trained well past the Chinchilla ratio for this reason.
A 2022 DeepMind finding that model size and training data should grow together. A common rule of thumb is roughly 20 training tokens per parameter.
They show smooth gains in prediction error as scale grows, but they are empirical fits, and gains in loss do not always match gains on a specific task.
Meta released Muse Wonder and Muse Wonder Full on Oct. 12, calling them its most capable models, available now through the Meta Model API and Muse Code. The post gives no scores, no prices and no safety report yet, so the frontier claim is Meta's until you test it.
Microsoft AI finished pre-training MAI-Base-2, a mixture-of-experts model with 2.8 trillion total and 80 billion active parameters, and says it is 4 to 12 times more compute-efficient than DeepSeek's base model. Nothing ships yet. The signal is that Microsoft wants its own frontier model.