Information theory is the mathematics of how much information a message carries and how efficiently it can be stored or sent. Claude Shannon founded the field in a 1948 paper written to answer an engineering question: how to send messages efficiently and reliably over noisy channels. The tools he built turned out to describe learning just as well, and they run through modern machine learning.
The central move is to measure information as surprise. An event you were sure of tells you nothing; an unlikely one tells you a lot. Averaging that surprise over all possible outcomes gives entropy, the minimum number of bits per symbol needed to encode messages from a source. The paper uses the word “bit” and credits it to John Tukey. It also proved that every noisy channel has a capacity, a maximum rate at which information can be sent with arbitrarily few errors.
Where it shows up in ML
- Cross-entropy: the standard loss for classification and for language models. It measures the average number of bits needed to encode the true answers with a code built from the model’s probabilities, so better predictions mean fewer bits.
- KL divergence: how far one probability distribution is from another. It appears in variational autoencoders, in knowledge distillation, and as the leash that keeps a model near its starting point during RLHF.
- Mutual information: how much knowing one variable tells you about another, used in feature selection and representation learning.
- Perplexity: the exponent of a language model’s cross-entropy, read as how many choices the model is effectively torn between at each step.
The link runs deeper than shared formulas. A model that predicts well can be used to compress well, and the reverse holds too, so training a language model to predict the next token is, in a precise sense, learning to compress text.