Learning When to Think Cuts Reasoning Tokens by 41 Percent

Reasoning models trained with reinforcement learning often burn the same token budget on easy arithmetic and hard proofs alike. A paper posted to arXiv on August 20, 2026 proposes a lighter approach: let the model choose how hard to think before it answers.

The work, titled Learning When to Think, trains a 1.5 billion parameter distilled model to pick one of three modes as its first output token. NoThink answers immediately. Short allows brief chain of thought. Long permits extended reasoning. The router lives inside Group Relative Policy Optimization rather than as a separate classifier, and a shaped reward makes each mode valuable at different response lengths.

Why adaptive compute matters

Test time scaling has become the default way labs squeeze performance from fixed weights. That helps on hard benchmarks, but it wastes capacity when a problem is trivial. The authors argue fixed budgets create over computation on easy items and under computation on difficult ones.

Their setup caps tokens per mode so the choices stay distinct. On MATH training data averaged across three seeds, the adaptive policy held accuracy near the base model on held out MATH500, 0.782 versus 0.796, while cutting mean response length from 4,796 tokens to 2,811, a 41 percent reduction.

Interestingly, the brief modes ended up more accurate than Long on many items, which suggests the router sorts problems by difficulty instead of defaulting to maximum thinking.

Transfer without retraining

The team evaluated zero shot transfer to other benchmarks. Savings were largest where problems skew easy. On GSM8K they reported a 76 percent token reduction while beating baselines at similar length. That pattern fits the hypothesis that a model can learn when extra reasoning pays off rather than always paying the full price.

The paper situates the method inside broader test time compute literature, including parallel work on when to stop thinking. The contribution here is an integrated GRPO trained gate with explicit mode caps rather than an external scheduler.

Limits and open questions

Results are on a small distilled model and math heavy tasks. Production reasoning stacks use far larger systems with tool access, retrieval, and safety filters. It remains unclear how well a three mode gate survives once models can call code interpreters or browse the web.

Still, the efficiency numbers are concrete. If adaptive routing survives scale up, it could reduce inference cost for consumer chat and agent loops where many turns are simple confirmations or lookups.

For researchers, the paper offers a reproducible recipe: encode mode choice as the first token, shape rewards so modes do not collapse, and measure both accuracy and tokens on held out suites.

Sources

arxiv.org/abs/2608.20256 Learning When to Think, submitted August 20, 2026

Authors include researchers from Tsinghua University and collaborating institutions per the arXiv entry