Pre-training is the first and by far the largest stage of building a modern model: train it on a huge amount of general data, with no particular task in mind, so it learns broadly useful patterns. What comes out is a base model. It is then adapted to specific jobs with much smaller, cheaper rounds of fine-tuning.
The idea is transfer learning at scale. Learning what language looks like, or what objects look like, is expensive and mostly the same whatever you later want to do. Do it once, then reuse it.
Where the labels come from
Pre-training works because it needs no human labelling. It uses self-supervised learning: the training signal is carved out of the raw data itself.
- Next-token prediction, the objective behind GPT-style models: read text left to right and predict each token from the ones before it. This is plain language modeling.
- Masked-token prediction, used by BERT in 2018: hide 15% of the tokens and predict them from the words on both sides.
Because any text can be turned into training examples this way, the data can be enormous. BERT pre-trained on about 3.3 billion words from books and English Wikipedia. GPT-3 was trained on 300 billion tokens.
Where it fits
The diagram shows why the split pays off. Pre-training is the expensive step, done once. The BERT paper notes that each of its fine-tuning results could be reproduced in at most an hour on a single Cloud TPU, starting from the same pre-trained model. ULMFiT, also from 2018, showed the data side: after language-model pre-training, a text classifier trained on 100 labelled examples matched one trained from scratch on 100 times more.
For chat assistants, “post-training” (instruction tuning, then RLHF or similar) turns the base model, which only continues text, into something that follows requests. Very large base models can also be steered with no fine-tuning at all, using examples in the prompt, which GPT-3 demonstrated.
The catch
A pre-trained model reflects its data: its gaps, its biases and its cut-off date. It predicts what is typical, not what is true, so it can continue a false claim as fluently as a true one. And because pre-training dominates the cost, decisions about model size and how many tokens to train on matter a great deal; that trade-off is what scaling laws try to predict.