[2602.04118] Learning to Reason in 13 Parameters

Abstract:Recent research has shown that language models can learn to \textit{reason}, often via reinforcement learning. Some work even trains low-rank parameterizations for reasoning, but conventional LoRA cannot scale below the model dimension. We question whether even rank=1 LoRA is necessary for learning to reason and propose TinyLoRA, a method for scaling low-rank adapters to sizes as small as one parameter. Within our new parameterization, we are able to train the 8B parameter size of Qwen2.5 to 91\% accuracy on GSM8K with only 13 trained parameters in bf16 (26 total bytes). We find this trend holds in general: we are able to recover 90\% of performance improvements while training $1000x$ fewer parameters across a suite of more difficult learning-to-reason benchmarks such as AIME, AMC, and MATH500. Notably, we are only able to achieve such strong performance with RL: models trained using SFT require $100-1000x$ larger updates to reach the same performance.

View PDF HTML (experimental) Abstract:Recent research has shown that language models can learn to \textit{reason}, often via reinforcement learning. Some work even trains low-rank parameterizations for reasoning, but conventional LoRA cannot scale below the model dimension. We question whether even rank=1 LoRA is necessary for learning to reason and propose TinyLoRA, a method for scaling low-rank adapters to sizes as small as one parameter. Within our new parameterization, we are able to train the 8B parameter size of Qwen2.5 to 91\% accuracy on GSM8K with only 13 trained parameters in bf16…

saved by

• Eric Liu

related reading

• LoRA Without Regret - Thinking Machines Labthinkingmachines.ai
• How We Build Trillion Parameter Reasoning RL with 10% GPUsmacaron.im
• [2106.09685] LoRA: Low-Rank Adaptation of Large Language Modelsarxiv.org
• DeepSeek-R1arxiv.org
• Explore | alphaXivalphaxiv.org
• LLM Resourcesforrestbicker.com
• The State of Reinforcement Learning for LLM Reasoningmagazine.sebastianraschka.com
• State of RL for reasoning LLMs | A. Weersaweers.de
• 2106.09685arxiv.org
• Pathways Language Model (PaLM): Scaling to 540 Billion Parameters for Breakthrouai.googleblog.com
• What Makes Low-Bit Quantization-Aware Training Work for Reasoning LLMs? A Systematic Studyarxiv.org
• Parameter-Efficient LLM Finetuning With Low-Rank Adaptation (LoRA) - Lightning AIlightning.ai