Putting Task Expertise into RL Achieves State-of-the-Art Performance on Text-to-SQL - Thinking Machines Lab
An expert-verified training set and a reshaped reward let RLVR on Tinker train a single model past human-level text-to-SQL accuracy, at a fraction of frontier cost.
Many industries rely on relational databases that are queried with SQL. Most SQL is machine-written, but humans alone likely write billions of custom SQL queries each monthBased on our internal estimates and publicly available data, such as from Snowflake filings. prompted by business questions. They are quite good at it — humans score 92.96% on BIRD, a realistic benchmark for translating natural-language questions into SQL. However, AI performance on text-to-SQL has lagged behind. LLM scores on the BIRD leaderboard improved from just below 70% in 2024 to 82% today. Frontier models like…
saved by
related reading
- Noisy Data Breaks RLVRddkang.substack.com
- Training a Misaligned Reward Seekeralignment.anthropic.com
- Learning to Replicate Expert Judgment in Financial Tasks - Thinking Machines Labthinkingmachines.ai
- Oversight Assistants: Turning Compute into Understandingbounded-regret.ghost.io
- BIRD-benchbird-bench.github.io
- Scaling Activation Oracles to Trillion-Parameter Modelstransluce.org
- The Extreme Inefficiency of RL for Frontier Models - Toby Ordtobyord.com
- [2510.13786] The Art of Scaling Reinforcement Learning Compute for LLMsarxiv.org
- [2504.13837] Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?arxiv.org
- [2005.11401] Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasksarxiv.org
- Language Models can Solve Computer Tasksarxiv.org
- Reasoning in General Domains without Verifiersarxiv.org