Training Mar 2026 · #96 most cited · 61 citations

Why Does Self-Distillation (Sometimes) Degrade the Reasoning Capability of LLMs?

Jeonghye Kim et al.

arXiv:2603.24472

In short

Self-distillation shortens reasoning, but in maths it can also make models worse, by up to 40%. The authors trace this to suppressing the model’s expressions of uncertainty while it reasons, which helps on familiar problems but hurts on new ones.

Why it matters

Shorter reasoning isn’t free: saying “I’m not sure” is part of how models reason well.

Read first

The 3 Field Guide ideas this paper leans on.

Starting from scratch? The full route 26 ideas · basics first
  1. Dataset ✓ understood

    A collection of data examples used for training, validating, or testing machine learning models.

  2. Training ✓ understood

    The process of fitting a model to data by repeatedly measuring how wrong its outputs are and adjusting its parameters to reduce that error.

  3. Machine Learning ✓ understood

    Building systems that learn patterns from data instead of following hand-written rules, getting better at a task as they see more examples.

  4. Neural Network ✓ understood

    A model built from layers of simple units, each taking a weighted sum of its inputs and applying a nonlinearity; training adjusts the weights so the whole stack maps inputs to useful outputs.

  5. Activation Function ✓ understood

    A non-linear function applied to neuron outputs that introduces non-linearity, enabling networks to learn complex patterns.

  6. Softmax ✓ understood

    A function that turns a list of scores (logits) into probabilities that are all positive and sum to 1; the standard output of classifiers and language models.

  7. Knowledge Distillation · read first ✓ understood

    Training a smaller 'student' model to mimic a larger 'teacher' model, transferring knowledge while reducing size.

  8. Natural Language Processing ✓ understood

    The field of AI that lets computers read, interpret, translate and generate human language, from spam filters and search to chatbots.

  9. Token ✓ understood

    The unit of text a language model reads and writes: a word, part of a word, or a character, drawn from the model's fixed vocabulary.

  10. Tokenization ✓ understood

    Splitting text into tokens, usually subword pieces, and mapping each to an integer ID so a language model can process it.

  11. Language Modeling ✓ understood

    Assigning probabilities to sequences of text, in practice by predicting each next token from the ones before it; the objective behind large language models.

  12. Unsupervised Learning ✓ understood

    Learning from unlabeled data to discover hidden patterns, structures, or relationships without explicit target outputs.

  13. Self-Supervised Learning ✓ understood

    Training on unlabeled data by making the data supply its own answers: hide or alter part of an input and have the model predict or match it.

  14. Pre-training ✓ understood

    Training a model on a large dataset (often self-supervised) before fine-tuning on specific tasks, enabling transfer learning.

  15. Feature ✓ understood

    A single measurable property of an example, such as a house's floor area or how many links an email contains, used as an input to a model.

  16. Deep Learning ✓ understood

    A subset of machine learning that uses neural networks with multiple layers (deep neural networks) to learn hierarchical representations of data.

  17. Representation Learning ✓ understood

    Letting a model learn its own internal description of the data, such as embeddings or layered visual features, instead of relying on features designed by hand.

  18. Embedding ✓ understood

    A list of numbers (a vector) that represents a word, sentence, image or other item, learned so that similar items end up close together.

  19. Attention Mechanism ✓ understood

    A technique that lets a neural network weigh every part of its input when producing each output, focusing on the parts most relevant at that step.

  20. Transformer ✓ understood

    A neural network architecture, introduced in 2017, built from stacked self-attention and feed-forward layers; the basis of nearly every modern large language model.

  21. Large Language Model ✓ understood

    A neural network, almost always a transformer, trained on vast amounts of text to predict the next token, which lets it write, answer, summarize and follow instructions.

  22. Prompt Engineering ✓ understood

    Designing a model's input (instructions, context, examples and output format) to get better results without changing the model itself.

  23. Chain-of-Thought · read first ✓ understood

    A prompting technique where the model explains its reasoning step-by-step before giving a final answer, improving complex reasoning.

  24. Supervised Learning ✓ understood

    Learning from examples paired with the correct answer, so a model can predict answers for new inputs it hasn't seen.

  25. Classification ✓ understood

    A supervised learning task where the model assigns each input to one of a fixed set of categories, such as spam or not spam.

  26. Calibration · read first ✓ understood

    Ensuring predicted probabilities accurately reflect true likelihood of outcomes.

In the frontier

Rank
#96 of 100
Citations
61
as of Aug 9, 2026
Published
Mar 2026

Topics: RL for reasoning , Reasoning methods , Interpretability and analysis

Selection: 1kpapers.com by Together AI, most-cited as of Aug 9, 2026

Nearby papers

Summary in our own words; read the paper for the details. ← All papers