Landmark 5 stops to get here · leads to 7

Pre-training

Training a model on a large dataset (often self-supervised) before fine-tuning on specific tasks, enabling transfer learning.

Your route here

5 stops · basics first
  1. Dataset ✓ understood

    A collection of data examples used for training, validating, or testing machine learning models.

  2. Training ✓ understood

    The process of fitting a model to data by repeatedly measuring how wrong its outputs are and adjusting its parameters to reduce that error.

  3. Machine Learning ✓ understood

    Building systems that learn patterns from data instead of following hand-written rules, getting better at a task as they see more examples.

  4. Unsupervised Learning ✓ understood

    Learning from unlabeled data to discover hidden patterns, structures, or relationships without explicit target outputs.

  5. Self-Supervised Learning ✓ understood

    Training on unlabeled data by making the data supply its own answers: hide or alter part of an input and have the model predict or match it.

  6. Pre-training · you are here ✓ understood

Picture it

  1. 01 Huge unlabeled dataset e.g. trillions of tokens of web text
  2. 02 Self-supervised objective Predict the next or the masked token
  3. 03 Base model General knowledge, no task focus yet
  4. 04 Fine-tuning Small labeled or task-specific dataset
  5. 05 Specialized model e.g. a chat assistant or classifier
Notice where the cost goes: pre-training does the expensive general learning once, so each fine-tune afterwards is cheap.

Pre-training is the first and by far the largest stage of building a modern model: train it on a huge amount of general data, with no particular task in mind, so it learns broadly useful patterns. What comes out is a base model. It is then adapted to specific jobs with much smaller, cheaper rounds of fine-tuning.

The idea is transfer learning at scale. Learning what language looks like, or what objects look like, is expensive and mostly the same whatever you later want to do. Do it once, then reuse it.

Where the labels come from

Pre-training works because it needs no human labelling. It uses self-supervised learning: the training signal is carved out of the raw data itself.

  • Next-token prediction, the objective behind GPT-style models: read text left to right and predict each token from the ones before it. This is plain language modeling.
  • Masked-token prediction, used by BERT in 2018: hide 15% of the tokens and predict them from the words on both sides.

Because any text can be turned into training examples this way, the data can be enormous. BERT pre-trained on about 3.3 billion words from books and English Wikipedia. GPT-3 was trained on 300 billion tokens.

Where it fits

The diagram shows why the split pays off. Pre-training is the expensive step, done once. The BERT paper notes that each of its fine-tuning results could be reproduced in at most an hour on a single Cloud TPU, starting from the same pre-trained model. ULMFiT, also from 2018, showed the data side: after language-model pre-training, a text classifier trained on 100 labelled examples matched one trained from scratch on 100 times more.

For chat assistants, “post-training” (instruction tuning, then RLHF or similar) turns the base model, which only continues text, into something that follows requests. Very large base models can also be steered with no fine-tuning at all, using examples in the prompt, which GPT-3 demonstrated.

The catch

A pre-trained model reflects its data: its gaps, its biases and its cut-off date. It predicts what is typical, not what is true, so it can continue a false claim as fluently as a true one. And because pre-training dominates the cost, decisions about model size and how many tokens to train on matter a great deal; that trade-off is what scaling laws try to predict.

Where it sits

Explore nearby

In the research

All papers →

4 papers that build on Pre-training .

Sources

  1. Devlin et al., "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding" . 2018
  2. Howard and Ruder, "Universal Language Model Fine-tuning for Text Classification" . 2018 (ULMFiT)
  3. Brown et al., "Language Models are Few-Shot Learners" . 2020 (GPT-3)