Landmark 5 stops to get here · leads to 7

Pre-training

Training a model on a large dataset (often self-supervised) before fine-tuning on specific tasks, enabling transfer learning.

Your route here

5 stops · basics first

Dataset ✓ understood
A collection of data examples used for training, validating, or testing machine learning models.
Training ✓ understood
The process of fitting a model to data by repeatedly measuring how wrong its outputs are and adjusting its parameters to reduce that error.
Machine Learning ✓ understood
Building systems that learn patterns from data instead of following hand-written rules, getting better at a task as they see more examples.
Unsupervised Learning ✓ understood
Learning from unlabeled data to discover hidden patterns, structures, or relationships without explicit target outputs.
Self-Supervised Learning ✓ understood
Learning representations from unlabeled data by creating supervised tasks from the data itself (masked prediction, contrastive learning).
Pre-training · you are here ✓ understood

Picture it

01 Huge unlabeled dataset e.g. trillions of tokens of web text
02 Self-supervised objective Predict the next or the masked token
03 Base model General knowledge, no task focus yet
04 Fine-tuning Small labeled or task-specific dataset
05 Specialized model e.g. a chat assistant or classifier

Notice where the cost goes: pre-training does the expensive general learning once, so each fine-tune afterwards is cheap.

Where it sits

Before this

Training Self-Supervised Learning

Pre-training

Leads to

Fine-Tuning Foundation Model Large Language Model Causal Language Modeling GPT Masked Language Modeling

+ 1 more

Explore nearby

Training Fine-Tuning The process of further training a pre-trained model on a specific dataset to adapt it for a particular task or domain. Training Transfer Learning Leveraging knowledge learned from one task/domain to improve performance on a related task with less data. Language & LLMs Foundation Model Large pre-trained models serving as a base for various downstream tasks (GPT, BERT, CLIP, SAM). Language & LLMs Language Modeling Learning probability distributions over sequences of words to predict what comes next. Language & LLMs Neural Scaling Laws Empirical relationships showing how model performance improves predictably with model size, data, and compute.

In the research

4 papers that build on Pre-training .

Canon · 2018 BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding It established “pre-train once, fine-tune everywhere”, and still powers much of search and classification.

Canon · 2022 Training Compute-Optimal Large Language Models It reset how labs size their models: data matters as much as parameter count.

Frontier · Oct 2025 · 104 citations Scaling Latent Reasoning via Looped Language Models Looping depth is a new scaling knob for reasoning that small models can exploit.

Frontier · Oct 2025 · 76 citations Poisoning Attacks on LLMs Require a Near-constant Number of Poison Samples Bigger training sets do not dilute poisoning, so attacks may be easier than assumed.