Recent LLMs can use filler tokens or problem repeats to improve (no-CoT) math performance
Prior results have shown that LLMs released before 2024 can’t leverage ‘filler tokens’—unrelated tokens prior to the model’s final answer—to perform additional computation and improve performance.1 I did an investigation on more recent models (e.g. Opus 4.5) and found that many recent LLMs improve substantially on math problems when given filler tokens. That is, I force the LLM to answer some math question immediately without being able to reason using Chain-of-Thought (CoT), but do give the LLM filter tokens (e.g., text like “Filler: 1 2 3 ...”) before it has to answer. Giving Opus 4.5 filler tokens2 boosts no-CoT performance from 45% to 51% (p=4e-7) on a dataset of relatively easy (competition) math problems3. I find a similar effect from repeating the problem statement many times, e.g. Opus 4.5’s no-CoT performance is boosted from 45% to 51%. Repeating the problem statement generally works better and is more reliable than filler tokens, especially for relatively weaker models I test
Prior results have shown that LLMs released before 2024 can’t leverage ‘filler tokens’—unrelated tokens prior to the model’s final answer—to perform additional computation and improve performance.1 I did an investigation on more recent models (e.g. Opus 4.5) and found that many recent LLMs improve substantially on math problems when given filler tokens. That is, I force the LLM to answer some math question immediately without being able to reason using Chain-of-Thought (CoT), but do give the LLM filler tokens (e.g., text like “Filler: 1 2 3 ...”) before it has to answer. Giving Opus 4.5…
saved by
related reading
- Measuring no CoT math time horizon (single forward pass)blog.redwoodresearch.org
- [2404.15758] Let's Think Dot by Dot: Hidden Computation in Transformer Language Modelsarxiv.org
- [2510.09714] All Code, No Thought: Current Language Models Struggle to Reason in Ciphered Languagearxiv.org
- Training Large Language Models to Reason in a Continuous Latent Spacearxiv.org
- Defeating Nondeterminism in LLM Inference - Thinking Machines Labthinkingmachines.ai
- Prefill awareness: can LLMs tell when “their” message history has been tampered with? — LessWronglesswrong.com
- LLMs are (mostly) not helped by filler tokens — LessWronglesswrong.com
- Can LLMs Be Computers?percepta.ai
- 2404.15758arxiv.org
- What sort of maths are LLMs good at?gowers.wordpress.com
- [2201.11903] Chain of Thought Prompting Elicits Reasoning in Large Language Modelsarxiv.org
- LLM Inference Performance Engineering: Best Practices | Databricks Blogdatabricks.com