Landmark 7 stops to get here

Foundation Model

A large model pre-trained on broad data at scale that can be adapted to many downstream tasks, such as GPT-style LLMs, BERT or CLIP.

Your route here

7 stops · basics first
  1. Dataset ✓ understood

    A collection of data examples used for training, validating, or testing machine learning models.

  2. Training ✓ understood

    The process of fitting a model to data by repeatedly measuring how wrong its outputs are and adjusting its parameters to reduce that error.

  3. Machine Learning ✓ understood

    Building systems that learn patterns from data instead of following hand-written rules, getting better at a task as they see more examples.

  4. Unsupervised Learning ✓ understood

    Learning from unlabeled data to discover hidden patterns, structures, or relationships without explicit target outputs.

  5. Self-Supervised Learning ✓ understood

    Training on unlabeled data by making the data supply its own answers: hide or alter part of an input and have the model predict or match it.

  6. Pre-training ✓ understood

    Training a model on a large dataset (often self-supervised) before fine-tuning on specific tasks, enabling transfer learning.

  7. Transfer Learning ✓ understood

    Leveraging knowledge learned from one task/domain to improve performance on a related task with less data.

  8. Foundation Model · you are here ✓ understood

Picture it

  1. Many downstream tasks Chat, search, coding, vision
  2. Adaptation Fine-tuning, prompting, adapters
  3. Foundation model e.g. GPT, BERT, CLIP
  4. Pre-training Self-supervised, very costly, done once
  5. Broad, massive data Web text, code, images
Reading upward, notice how one expensive pre-trained base fans out to many tasks through cheap adaptation.

A foundation model is a large model trained once on broad data, at scale, and then adapted to many different tasks. Instead of building a separate model for translation, summarization and question answering, you start from one pre-trained base and specialize it with a little extra training, or just a well-written prompt.

The term comes from a 2021 report by Rishi Bommasani, Percy Liang and colleagues at Stanford’s Center for Research on Foundation Models. Pointing to models such as BERT, DALL-E and GPT-3, they chose “foundation” to stress that these models are central but incomplete: a great deal gets built on top of them, and none is a finished product on its own.

How it works

The diagram reads bottom to top. Pre-training on web text, code or image–caption pairs is usually self-supervised: the data supplies its own training signal, such as the next word or which caption goes with which image, so no one has to hand-label data at that scale. This stage is the expensive one. Adaptation is comparatively cheap:

  • Fine-tuning on a smaller, task-specific dataset, either the whole model or a lightweight add-on such as LoRA.
  • Prompting, including a few worked examples, with no training at all.
  • Using the representations directly, as when CLIP embeddings power image search.

The report is candid that the ingredients aren’t new: this is deep learning plus transfer learning. What changed is scale.

Emergence and homogenization

The report organizes its argument around two ideas. Emergence: at scale, models show capabilities nobody explicitly built in. Homogenization: a huge range of applications ends up resting on a handful of base models. That concentration is efficient, since one improvement to the base helps every product built on it. It also cuts the other way: whatever is wrong with the foundation model is inherited by everything adapted from it.

The catch

A foundation model’s biases, blind spots and failure modes travel into every product built on top, often unnoticed, because each team sees only its own slice. Training one takes compute few organizations can afford, which limits who gets to build them. And the report’s own warning is worth keeping in mind: for all their reach, we lack a clear understanding of how these models work, when they fail, and what they’re capable of.

Where it sits

Foundation Model

Leads to

Nothing yet: a destination in its own right.

Explore nearby

In the research

All papers →

2 papers that build on Foundation Model .

Sources

  1. Bommasani et al., "On the Opportunities and Risks of Foundation Models" . Stanford Center for Research on Foundation Models, 2021