Language modeling is the task of assigning a probability to a sequence of text. In practice that means one question, asked over and over: given the words so far, how likely is each possible next word? A model that answers it well has to pick up spelling, grammar, facts and some reasoning along the way, because all of them help predict what comes next.
The trick that makes this tractable is the chain rule of probability. The probability of a whole sentence equals the probability of the first word, times the probability of the second given the first, and so on. So a model that can predict one step ahead can score, or write, a text of any length.
How it works
The diagram shows the loop. Text is split into tokens, and the model reads the context and outputs a probability for every token in its vocabulary, usually through a softmax. Generation picks one token, appends it and runs again. Whether it picks the single likeliest token or samples, and how adventurously, is a decoding choice (temperature is the usual dial), not something the model itself decides.
Training is the mirror image. Take real text, hide what comes next, and adjust the model to put more probability on the token that actually appeared. No human labels are needed, which is why language modeling scales to web-sized data and why it is the objective behind pre-training.
From counting to transformers
- N-gram models assume only the last few words matter and estimate probabilities by counting. They are fast, but most long word sequences never appear in any corpus, so counts run out.
- Neural language models, proposed by Bengio and colleagues in 2003, learn a vector for each word, so a sentence never seen before can still score well if it resembles ones that were.
- Transformer models condition on thousands of previous tokens. Scaled up, they become large language models; GPT-3 had 175 billion parameters.
A causal model predicts left to right and can generate. A masked model like BERT fills in hidden words using context on both sides, which suits understanding tasks rather than writing.
The catch
The standard score is perplexity: roughly, how surprised the model is by held-out text. Lower is better, but it measures prediction, not truth. A model can assign high probability to a fluent, confident sentence that is false, because its objective rewards what is likely to be written, not what is correct.