Landmark 3 stops to get here · leads to 8

Language Modeling

Assigning probabilities to sequences of text, in practice by predicting each next token from the ones before it; the objective behind large language models.

Your route here

3 stops · basics first
  1. Natural Language Processing ✓ understood

    The field of AI that lets computers read, interpret, translate and generate human language, from spam filters and search to chatbots.

  2. Token ✓ understood

    The unit of text a language model reads and writes: a word, part of a word, or a character, drawn from the model's fixed vocabulary.

  3. Tokenization ✓ understood

    Splitting text into tokens, usually subword pieces, and mapping each to an integer ID so a language model can process it.

  4. Language Modeling · you are here ✓ understood

Picture it

  1. 01 Context "The cat sat on the"
  2. 02 Language model Reads all previous tokens
  3. 03 Next-token distribution P(mat) 0.41, P(floor) 0.22, P(sofa) 0.09…
  4. 04 Pick a token Greedy or sampled: "mat"
  5. 05 Append and repeat The new token joins the context
Notice the model never outputs a sentence directly: it outputs a probability for every possible next token, one step at a time.

Language modeling is the task of assigning a probability to a sequence of text. In practice that means one question, asked over and over: given the words so far, how likely is each possible next word? A model that answers it well has to pick up spelling, grammar, facts and some reasoning along the way, because all of them help predict what comes next.

The trick that makes this tractable is the chain rule of probability. The probability of a whole sentence equals the probability of the first word, times the probability of the second given the first, and so on. So a model that can predict one step ahead can score, or write, a text of any length.

How it works

The diagram shows the loop. Text is split into tokens, and the model reads the context and outputs a probability for every token in its vocabulary, usually through a softmax. Generation picks one token, appends it and runs again. Whether it picks the single likeliest token or samples, and how adventurously, is a decoding choice (temperature is the usual dial), not something the model itself decides.

Training is the mirror image. Take real text, hide what comes next, and adjust the model to put more probability on the token that actually appeared. No human labels are needed, which is why language modeling scales to web-sized data and why it is the objective behind pre-training.

From counting to transformers

  • N-gram models assume only the last few words matter and estimate probabilities by counting. They are fast, but most long word sequences never appear in any corpus, so counts run out.
  • Neural language models, proposed by Bengio and colleagues in 2003, learn a vector for each word, so a sentence never seen before can still score well if it resembles ones that were.
  • Transformer models condition on thousands of previous tokens. Scaled up, they become large language models; GPT-3 had 175 billion parameters.

A causal model predicts left to right and can generate. A masked model like BERT fills in hidden words using context on both sides, which suits understanding tasks rather than writing.

The catch

The standard score is perplexity: roughly, how surprised the model is by held-out text. Lower is better, but it measures prediction, not truth. A model can assign high probability to a fluent, confident sentence that is false, because its objective rewards what is likely to be written, not what is correct.

Where it sits

Explore nearby

In the research

All papers →

4 papers that build on Language Modeling .

Sources

  1. Daniel Jurafsky and James H. Martin, Speech and Language Processing, chapter 3: N-gram Language Models . 3rd edition draft, 2026
  2. Bengio et al., "A Neural Probabilistic Language Model" . Journal of Machine Learning Research, 2003
  3. Brown et al., "Language Models are Few-Shot Learners" . 2020 (GPT-3)