A hyperparameter in knowledge distillation controlling how soft the teacher's outputs are.
A collection of data examples used for training, validating, or testing machine learning models.
The process of fitting a model to data by repeatedly measuring how wrong its outputs are and adjusting its parameters to reduce that error.
Building systems that learn patterns from data instead of following hand-written rules, getting better at a task as they see more examples.
A computational model inspired by biological neural networks, consisting of interconnected nodes (neurons) organized in layers that process information through weighted connections.
A non-linear function applied to neuron outputs that introduces non-linearity, enabling networks to learn complex patterns.
A function that turns a list of scores (logits) into probabilities that are all positive and sum to 1; the standard output of classifiers and language models.
Training a smaller 'student' model to mimic a larger 'teacher' model, transferring knowledge while reducing size.
A parameter controlling the smoothness of probability distributions in softmax - lower makes peaks sharper, higher makes it more uniform.
A hyperparameter in knowledge distillation controlling how soft the teacher’s outputs are.
This concept is essential for understanding training & optimization and forms a key part of modern AI systems.
Leads to
Nothing yet: a destination in its own right.