Training Aug 2025 · #100 most cited · 59 citations

TreePO: Bridging the Gap of Policy Optimization and Efficacy and Inference Efficiency with Heuristic Tree-based Modeling

Yizhi Li et al.

arXiv:2508.17445

In short

TreePO samples reasoning chains as a tree: shared beginnings are generated once and branches split where the model is uncertain, with weak branches pruned early. That cuts RL sampling compute by up to 43% while keeping or improving exploration.

Why it matters

RL post-training is expensive; sharing work across rollouts makes it cheaper.

Read first

The 3 Field Guide ideas this paper leans on.

Agents & RL · Landmark Reinforcement Learning Learning through interaction with an environment, receiving rewards or penalties to learn optimal behavior policies. Agents & RL · Standard Policy Gradient RL methods that directly optimize the policy by computing gradients of expected reward with respect to policy parameters. Language & LLMs · Standard Beam Search A generation algorithm that maintains top-k candidates at each step, balancing quality and diversity.

Starting from scratch? The full route 16 ideas · basics first

Machine Learning ✓ understood
Building systems that learn patterns from data instead of following hand-written rules, getting better at a task as they see more examples.
Reinforcement Learning · read first ✓ understood
Learning through interaction with an environment, receiving rewards or penalties to learn optimal behavior policies.
Agent ✓ understood
In RL, the learner or decision-maker that takes actions in an environment to maximize cumulative reward.
Policy ✓ understood
A strategy or mapping from states to actions that defines the agent's behavior in reinforcement learning.
Dataset ✓ understood
A collection of data examples used for training, validating, or testing machine learning models.
Training ✓ understood
The process of fitting a model to data by repeatedly measuring how wrong its outputs are and adjusting its parameters to reduce that error.
Loss Function ✓ understood
A function that scores how wrong a model's prediction is as a single number, which training then works to make as small as possible.
Gradient Descent ✓ understood
An optimization method that repeatedly moves a model's parameters a small step in the direction that most reduces the loss.
Policy Gradient · read first ✓ understood
RL methods that directly optimize the policy by computing gradients of expected reward with respect to policy parameters.
Natural Language Processing ✓ understood
The field of AI that lets computers read, interpret, translate and generate human language, from spam filters and search to chatbots.
Token ✓ understood
The basic unit of text that a language model processes, typically representing a word, subword, or character. Tokens are the fundamental building blocks for LLM input and output.
Tokenization ✓ understood
Splitting text into tokens, usually subword pieces, and mapping each to an integer ID so a language model can process it.
Language Modeling ✓ understood
Learning probability distributions over sequences of words to predict what comes next.
Autoregressive Model ✓ understood
A model that generates output one token at a time, using previously generated tokens as input for the next prediction.
Greedy Decoding ✓ understood
Always selecting the most likely next token during generation, fast but can lead to repetitive or suboptimal outputs.
Beam Search · read first ✓ understood
A generation algorithm that maintains top-k candidates at each step, balancing quality and diversity.

In the frontier

Rank: #100 of 100
Citations: 59; as of Aug 9, 2026
Published: Aug 2025

Topics: RL for reasoning , Efficiency and serving

Selection: 1kpapers.com by Together AI, most-cited as of Aug 9, 2026

Nearby papers

Canon · 2013 Playing Atari with Deep Reinforcement Learning It started deep reinforcement learning: one learner, raw perception, many tasks. Canon · 2016 Mastering the Game of Go with Deep Neural Networks and Tree Search It proved that learned intuition plus search can master a problem long thought a decade away. Canon · 2017 Proximal Policy Optimization Algorithms PPO became the default RL algorithm, including for RLHF on the first ChatGPT-era models. Canon · 2022 Training Language Models to Follow Instructions with Human Feedback This recipe, RLHF, is what turned raw language models into assistants like ChatGPT.