Standard 4 stops to get here

Next-Token Prediction

The task of predicting the next token in a sequence from all the tokens before it: the training objective behind GPT-style language models and how they generate text.

Your route here

4 stops · basics first
  1. Token ✓ understood

    The unit of text a language model reads and writes: a word, part of a word, or a character, drawn from the model's fixed vocabulary.

  2. Natural Language Processing ✓ understood

    The field of AI that lets computers read, interpret, translate and generate human language, from spam filters and search to chatbots.

  3. Tokenization ✓ understood

    Splitting text into tokens, usually subword pieces, and mapping each to an integer ID so a language model can process it.

  4. Language Modeling ✓ understood

    Assigning probabilities to sequences of text, in practice by predicting each next token from the ones before it; the objective behind large language models.

  5. Next-Token Prediction · you are here ✓ understood

Picture it

  1. 01 Tokens so far "The cat sat on the"
  2. 02 Model Reads the context, scores every token
  3. 03 Probabilities "mat" likely, "floor" less so, "moon" unlikely
  4. 04 Train or generate Score the real next token, or pick one
The same step does both jobs: in training it is graded against the real next token; in use its pick is appended and the loop repeats.

Next-token prediction is the task of guessing what comes next in a sequence, given everything before it. Shown “The cat sat on the”, a model outputs a probability for every token in its vocabulary, with “mat” and “floor” near the top. It is the objective that GPT-style large language models are pre-trained on, and the loop they run to generate text.

As a training objective

Text supplies its own answers: every position in a document is a free example of “given this, what came next?” That makes next-token prediction a form of self-supervised learning that can use enormous amounts of unlabeled text. The model is scored with cross-entropy loss, the negative log of the probability it gave the actual next token, and a causal mask makes sure it can’t peek ahead. The formal name for this setup is causal language modeling.

The idea predates transformers. Bengio and colleagues trained a neural network to predict the next word in 2003, learning word embeddings along the way. GPT-3 showed that scaling the same objective up to a 175-billion-parameter model produced a system that could perform many tasks from a few examples in its prompt.

As a way to generate

At inference time, the model predicts a distribution, one token is chosen from it, the token is appended to the input, and the process repeats. How that token is chosen is up to the decoding strategy, from always taking the most likely token to sampling with a temperature. This one-at-a-time loop is why output streams word by word and why longer answers take longer.

The catch

The objective rewards plausible continuations, not true ones. A model trained only to predict the next token will confidently continue a false premise, which is one root of hallucination and why further training such as instruction tuning and RLHF follows pre-training.

Where it sits

Before this

Language Modeling Token
Next-Token Prediction

Leads to

Nothing yet: a destination in its own right.

Explore nearby

Language & LLMs Causal Language Modeling Training a model to predict the next token given previous tokens, the foundation of autoregressive models like GPT. Language & LLMs Autoregressive Model A model that generates output one token at a time, using previously generated tokens as input for the next prediction. Training Pre-training Training a model on a large dataset (often self-supervised) before fine-tuning on specific tasks, enabling transfer learning. Training Cross-Entropy Loss A loss function for classification that measures the difference between predicted and true probability distributions. Evaluation Perplexity A metric measuring how well a language model predicts text - lower perplexity indicates better prediction. Neural Networks Softmax A function that turns a list of scores (logits) into probabilities that are all positive and sum to 1; the standard output of classifiers and language models.

Sources

  1. Yoshua Bengio, Réjean Ducharme, Pascal Vincent and Christian Jauvin, "A Neural Probabilistic Language Model" . Journal of Machine Learning Research 3, 2003
  2. Tom B. Brown et al., "Language Models are Few-Shot Learners" . NeurIPS 2020; GPT-3