Definition
Pretraining is Pretraining is the first and largest stage of building an AI model, where it learns general patterns by predicting the next token across enormous amounts of text, code and other data.
Pretraining is where a model gets its general knowledge. Before a model can chat, write code or follow instructions, it goes through a long, expensive first phase of reading, in a sense, a huge slice of text, code and sometimes images, audio and video.
The task is simple to state: predict what comes next. The model sees a stretch of text with the next token hidden and guesses it. A wrong guess nudges its parameters, the learned numbers inside it, so the next guess is slightly better. Repeat this trillions of times and the model picks up grammar, facts, styles, reasoning patterns and how code is structured, all without anyone labeling the data.
The result is a base model. It is a powerful text predictor, but not yet a helpful assistant. Ask it a question and it may continue with more questions, because that is a plausible continuation.
Pretraining is the most compute-hungry step in building a model. It needs large clusters of specialized chips running for weeks, plus vast amounts of cleaned data. The scale of the data and the compute, along with the model's size, largely determine how capable the base model can become. This is why only a small number of organizations pretrain the biggest models from scratch.
Designs such as mixture of experts are partly attempts to get more capability from a given pretraining budget.
A base model becomes a product through post-training: fine-tuning on examples of good answers, then reinforcement learning from feedback and rewards. Pretraining supplies the knowledge. Post-training shapes how the model uses it.
Pretraining is done once, by the lab, ahead of release. Inference is every time someone uses the finished model. When a model is said to have a "knowledge cutoff," that cutoff comes from the end of its pretraining data.
The first, largest stage of building a model, where it learns general language and knowledge by predicting the next token across huge amounts of data before any task-specific tuning.
Pretraining builds broad general knowledge from massive data. Fine-tuning then adjusts the pretrained model on a smaller, targeted set of examples for a specific behavior or task.
Reflection AI introduced Beam on Monday, a 501-billion-parameter mixture-of-experts model with 23 billion active, and promised Apache 2.0 weights later in October. Until the weights ship, it is a waitlist and a claim about efficiency.
Nvidia is in talks to acquire open-weight model startup Reflection AI or invest more in it, the Financial Times reported, days after Reflection announced its 501B-parameter Beam model. Nothing changes for you until Beam's weights ship under Apache 2.0.