Landmark 18 stops to get here · leads to 1

Retrieval-Augmented Generation

Fetching relevant documents at question time and adding them to an LLM's prompt, so answers can draw on current, checkable sources without retraining.

Your route here

18 stops · basics first
  1. Dataset ✓ understood

    A collection of data examples used for training, validating, or testing machine learning models.

  2. Feature ✓ understood

    A single measurable property of an example, such as a house's floor area or how many links an email contains, used as an input to a model.

  3. Machine Learning ✓ understood

    Building systems that learn patterns from data instead of following hand-written rules, getting better at a task as they see more examples.

  4. Neural Network ✓ understood

    A model built from layers of simple units, each taking a weighted sum of its inputs and applying a nonlinearity; training adjusts the weights so the whole stack maps inputs to useful outputs.

  5. Deep Learning ✓ understood

    A subset of machine learning that uses neural networks with multiple layers (deep neural networks) to learn hierarchical representations of data.

  6. Representation Learning ✓ understood

    Letting a model learn its own internal description of the data, such as embeddings or layered visual features, instead of relying on features designed by hand.

  7. Embedding ✓ understood

    A list of numbers (a vector) that represents a word, sentence, image or other item, learned so that similar items end up close together.

  8. Natural Language Processing ✓ understood

    The field of AI that lets computers read, interpret, translate and generate human language, from spam filters and search to chatbots.

  9. Token ✓ understood

    The unit of text a language model reads and writes: a word, part of a word, or a character, drawn from the model's fixed vocabulary.

  10. Tokenization ✓ understood

    Splitting text into tokens, usually subword pieces, and mapping each to an integer ID so a language model can process it.

  11. Language Modeling ✓ understood

    Assigning probabilities to sequences of text, in practice by predicting each next token from the ones before it; the objective behind large language models.

  12. Training ✓ understood

    The process of fitting a model to data by repeatedly measuring how wrong its outputs are and adjusting its parameters to reduce that error.

  13. Unsupervised Learning ✓ understood

    Learning from unlabeled data to discover hidden patterns, structures, or relationships without explicit target outputs.

  14. Self-Supervised Learning ✓ understood

    Training on unlabeled data by making the data supply its own answers: hide or alter part of an input and have the model predict or match it.

  15. Pre-training ✓ understood

    Training a model on a large dataset (often self-supervised) before fine-tuning on specific tasks, enabling transfer learning.

  16. Attention Mechanism ✓ understood

    A technique that lets a neural network weigh every part of its input when producing each output, focusing on the parts most relevant at that step.

  17. Transformer ✓ understood

    A neural network architecture, introduced in 2017, built from stacked self-attention and feed-forward layers; the basis of nearly every modern large language model.

  18. Large Language Model ✓ understood

    A neural network, almost always a transformer, trained on vast amounts of text to predict the next token, which lets it write, answer, summarize and follow instructions.

  19. Retrieval-Augmented Generation · you are here ✓ understood

Picture it

  1. 01 User query
  2. 02 Embed the query Turn it into a vector
  3. 03 Retrieve documents Nearest matches from a vector database
  4. 04 Augment the prompt Query + retrieved passages together
  5. 05 LLM generates answer Grounded in the retrieved text
Notice the model's weights never change: fresh knowledge comes in through the prompt, fetched at the moment of the question.

Retrieval-augmented generation (RAG) gives a language model an open book. Before the model answers, a retriever searches a collection of documents for passages relevant to the question and places them in the prompt. The model then writes its answer from that text rather than from memory alone.

The name comes from a 2020 paper by Lewis and colleagues. They paired a pre-trained sequence-to-sequence generator with a dense vector index of Wikipedia and a neural retriever, describing it as combining parametric memory (knowledge stored in the weights) with non-parametric memory (an index you can read and replace). Their models set the state of the art on three open-domain question-answering tasks. (Paper summary.)

How a RAG pipeline works

The diagram shows the path at question time. Ahead of time, documents are split into chunks, each chunk is turned into an embedding, and the vectors are stored in a vector database. Then, for each question:

  1. The question is embedded with the same model.
  2. The database returns the chunks whose vectors are nearest, a stand-in for semantic similarity.
  3. The chunks and the question go into the prompt together.
  4. The model answers, ideally saying which chunk each claim came from.

Some systems add keyword search alongside the vector search, or a reranking step that rescores candidates before they reach the prompt.

Why use it

  • Fresh knowledge without retraining. Update the documents and the answers change. The weights never do.
  • Private data. A company’s own documents can be searched at question time without being trained into a model.
  • Checkable answers. Retrieved passages give provenance, which the 2020 paper flagged as an open problem for models that keep knowledge only in their weights. Grounding answers in sources also tends to reduce hallucination, though it doesn’t remove it.

The catch

RAG is only as good as its retrieval. If the right passage isn’t found, the model answers from memory or from a near-miss chunk, and sounds just as sure. Chunking is a real design choice: split too finely and context is lost, too coarsely and irrelevant text crowds the prompt. Retrieved text also uses up the context window, and every token costs money, so more passages isn’t automatically better. And the model can still misread or ignore a good source, which is why retrieval and answers are worth evaluating separately.

Where it sits

Before this

Large Language Model Embedding
Retrieval-Augmented Generation

Leads to

Retrieval-Interleaved Generation

Explore nearby

Language & LLMs Information Retrieval The field of finding the documents in a large collection that satisfy a person's information need, and ranking them by how relevant they are. Language & LLMs Vector Database A database optimized for storing and searching high-dimensional vectors (embeddings), enabling semantic search and RAG. Language & LLMs Hallucination When a language model produces fluent, confident output that is false or unsupported by its sources, such as invented facts, quotes or citations. Language & LLMs Grounding Connecting model outputs to real-world facts, sources, or evidence to improve factuality and reduce hallucinations. Language & LLMs Retrieval-Interleaved Generation Dynamically retrieving information during generation rather than just before, allowing models to gather facts as needed. Language & LLMs Context Window The maximum number of tokens a language model can take into account at once, counting both the input it reads and the output it writes. Also called context length. Language & LLMs Semantic Similarity Measuring how similar two pieces of text are in meaning, often using embedding-based distance metrics.

In the research

All papers →

2 papers that build on Retrieval-Augmented Generation .

Sources

  1. Lewis et al., "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks" . 2020