A foundation model is a large model trained once on broad data, at scale, and then adapted to many different tasks. Instead of building a separate model for translation, summarization and question answering, you start from one pre-trained base and specialize it with a little extra training, or just a well-written prompt.
The term comes from a 2021 report by Rishi Bommasani, Percy Liang and colleagues at Stanford’s Center for Research on Foundation Models. Pointing to models such as BERT, DALL-E and GPT-3, they chose “foundation” to stress that these models are central but incomplete: a great deal gets built on top of them, and none is a finished product on its own.
How it works
The diagram reads bottom to top. Pre-training on web text, code or image–caption pairs is usually self-supervised: the data supplies its own training signal, such as the next word or which caption goes with which image, so no one has to hand-label data at that scale. This stage is the expensive one. Adaptation is comparatively cheap:
- Fine-tuning on a smaller, task-specific dataset, either the whole model or a lightweight add-on such as LoRA.
- Prompting, including a few worked examples, with no training at all.
- Using the representations directly, as when CLIP embeddings power image search.
The report is candid that the ingredients aren’t new: this is deep learning plus transfer learning. What changed is scale.
Emergence and homogenization
The report organizes its argument around two ideas. Emergence: at scale, models show capabilities nobody explicitly built in. Homogenization: a huge range of applications ends up resting on a handful of base models. That concentration is efficient, since one improvement to the base helps every product built on it. It also cuts the other way: whatever is wrong with the foundation model is inherited by everything adapted from it.
The catch
A foundation model’s biases, blind spots and failure modes travel into every product built on top, often unnoticed, because each team sees only its own slice. Training one takes compute few organizations can afford, which limits who gets to build them. And the report’s own warning is worth keeping in mind: for all their reach, we lack a clear understanding of how these models work, when they fail, and what they’re capable of.