Standard 9 stops to get here · leads to 4

Decoding Strategies

The rules a language model uses to turn its next-token probabilities into actual text, from always taking the top token to sampling with temperature, top-k or top-p.

Your route here

9 stops · basics first
  1. Natural Language Processing ✓ understood

    The field of AI that lets computers read, interpret, translate and generate human language, from spam filters and search to chatbots.

  2. Token ✓ understood

    The unit of text a language model reads and writes: a word, part of a word, or a character, drawn from the model's fixed vocabulary.

  3. Tokenization ✓ understood

    Splitting text into tokens, usually subword pieces, and mapping each to an integer ID so a language model can process it.

  4. Language Modeling ✓ understood

    Assigning probabilities to sequences of text, in practice by predicting each next token from the ones before it; the objective behind large language models.

  5. Autoregressive Model ✓ understood

    A model that generates output one token at a time, using previously generated tokens as input for the next prediction.

  6. Machine Learning ✓ understood

    Building systems that learn patterns from data instead of following hand-written rules, getting better at a task as they see more examples.

  7. Neural Network ✓ understood

    A model built from layers of simple units, each taking a weighted sum of its inputs and applying a nonlinearity; training adjusts the weights so the whole stack maps inputs to useful outputs.

  8. Activation Function ✓ understood

    A non-linear function applied to neuron outputs that introduces non-linearity, enabling networks to learn complex patterns.

  9. Softmax ✓ understood

    A function that turns a list of scores (logits) into probabilities that are all positive and sum to 1; the standard output of classifiers and language models.

  10. Decoding Strategies · you are here ✓ understood

Picture it

  1. 01 Logits A raw score for every token in the vocabulary
  2. 02 Temperature Sharpen or flatten the distribution
  3. 03 Truncate Top-k or top-p drops the unlikely tail
  4. 04 Pick one token Take the top choice, or sample
Each stage is a knob. Greedy decoding skips straight to the top choice; sampling methods reshape and trim the distribution first.

A language model doesn’t output text directly. At each step it outputs a probability for every token in its vocabulary, and a decoding strategy decides which token to actually emit. The same model can write flat, repetitive prose or varied, fluent text depending only on this choice.

Search: aim for the most likely text

  • Greedy decoding always takes the single most likely token. It is fast and deterministic, but it can miss a better sentence that starts with a less likely word.
  • Beam search keeps several candidate sequences alive and returns the one with the highest overall probability. It suits tasks with one right answer, such as translation.

Sampling: draw from the distribution

  • Temperature rescales the probabilities before sampling: below 1 makes likely tokens more likely, above 1 flattens the choice.
  • Top-k samples only from the k most likely tokens.
  • Top-p, or nucleus sampling, samples from the smallest set of tokens whose probabilities add up to p, so the shortlist grows when the model is unsure and shrinks when it is confident.

Repetition penalties, stop sequences and length limits are layered on top.

Why the most likely text isn’t the best

Holtzman et al. found that maximizing likelihood during decoding produces text that is “bland and strangely repetitive”, even from a strong model. Human writing is not made of the most probable word at every step. Sampling while cutting off the unreliable low-probability tail, which is what top-p does, produced text closer to human writing. That is why chat assistants generally sample, while tasks that want a single exact answer lean on greedy or beam search.

Where it sits

Before this

Autoregressive Model Softmax
Decoding Strategies

Leads to

Beam Search Greedy Decoding Top-k Sampling Top-p Sampling

Explore nearby

Language & LLMs Greedy Decoding Always selecting the most likely next token during generation, fast but can lead to repetitive or suboptimal outputs. Language & LLMs Beam Search A generation algorithm that maintains top-k candidates at each step, balancing quality and diversity. Language & LLMs Top-k Sampling A generation strategy that samples from only the k most likely next tokens, balancing quality and diversity. Language & LLMs Top-p Sampling Nucleus sampling - selecting from the smallest set of tokens whose cumulative probability exceeds p, providing dynamic vocabulary. Language & LLMs Temperature A sampling parameter controlling randomness in generation - lower values make output more deterministic, higher more creative. Language & LLMs Repetition Penalty A technique reducing the likelihood of previously generated tokens to avoid repetitive outputs.

Sources

  1. Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes and Yejin Choi, "The Curious Case of Neural Text Degeneration" . ICLR 2020; introduces nucleus (top-p) sampling
  2. Angela Fan, Mike Lewis and Yann Dauphin, "Hierarchical Neural Story Generation" . ACL 2018; top-k sampling
  3. Patrick von Platen, "How to generate text" . Hugging Face blog, 2020 (updated 2023)