Landmark 7 stops to get here

Foundation Model

Large pre-trained models serving as a base for various downstream tasks (GPT, BERT, CLIP, SAM).

Your route here

7 stops · basics first
  1. Dataset ✓ understood

    A collection of data examples used for training, validating, or testing machine learning models.

  2. Training ✓ understood

    The process of fitting a model to data by repeatedly measuring how wrong its outputs are and adjusting its parameters to reduce that error.

  3. Machine Learning ✓ understood

    Building systems that learn patterns from data instead of following hand-written rules, getting better at a task as they see more examples.

  4. Unsupervised Learning ✓ understood

    Learning from unlabeled data to discover hidden patterns, structures, or relationships without explicit target outputs.

  5. Self-Supervised Learning ✓ understood

    Learning representations from unlabeled data by creating supervised tasks from the data itself (masked prediction, contrastive learning).

  6. Pre-training ✓ understood

    Training a model on a large dataset (often self-supervised) before fine-tuning on specific tasks, enabling transfer learning.

  7. Transfer Learning ✓ understood

    Leveraging knowledge learned from one task/domain to improve performance on a related task with less data.

  8. Foundation Model · you are here ✓ understood

Picture it

  1. Many downstream tasks Chat, search, coding, vision
  2. Adaptation Fine-tuning, prompting, adapters
  3. Foundation model e.g. GPT, BERT, CLIP
  4. Pre-training Self-supervised, very costly, done once
  5. Broad, massive data Web text, code, images
Reading upward, notice how one expensive pre-trained base fans out to many tasks through cheap adaptation.

Where it sits

Before this

Pre-training Transfer Learning
Foundation Model

Leads to

Nothing yet: a destination in its own right.

Explore nearby

Language & LLMs Large Language Model A neural network, almost always a transformer, trained on vast amounts of text to predict the next token, which lets it write, answer, summarize and follow instructions. Vision & Multimodal CLIP Contrastive Language-Image Pre-training - a model jointly trained on images and text, enabling zero-shot image classification. Training Fine-Tuning The process of further training a pre-trained model on a specific dataset to adapt it for a particular task or domain. Language & LLMs Neural Scaling Laws Empirical relationships showing how model performance improves predictably with model size, data, and compute.

In the research

All papers →

2 papers that build on Foundation Model .