Standard A starting point · leads to 2

Entropy

A measure of uncertainty or randomness in a random variable from information theory.

A measure of uncertainty or randomness in a random variable from information theory.

This concept is essential for understanding machine learning fundamentals and forms a key part of modern AI systems.

Information Theory
Cross-Entropy
KL Divergence

Where it sits

Before this

Nothing: this is a starting point.

Entropy

Leads to

KL Divergence Mutual Information

Explore nearby

Training Cross-Entropy Loss A loss function for classification that measures the difference between predicted and true probability distributions. Foundations KL Divergence Kullback-Leibler divergence - a measure of how one probability distribution differs from another. Foundations Mutual Information A measure of dependence between variables, quantifying how much knowing one reduces uncertainty about the other. Foundations Information Bottleneck A principle for learning representations that compress input while retaining information relevant to prediction. Evaluation Perplexity A metric measuring how well a language model predicts text - lower perplexity indicates better prediction.