Putting Task Expertise into RL Achieves State-of-the-Art Performance on Text-to-SQL - Thinking Machines Lab

thinkingmachines.ai · 2,568 words · saved by 3 readers

An expert-verified training set and a reshaped reward let RLVR on Tinker train a single model past human-level text-to-SQL accuracy, at a fraction of frontier cost.

Many industries rely on relational databases that are queried with SQL. Most SQL is machine-written, but humans alone likely write billions of custom SQL queries each monthBased on our internal estimates and publicly available data, such as from Snowflake filings. prompted by business questions. They are quite good at it — humans score 92.96% on BIRD, a realistic benchmark for translating natural-language questions into SQL. However, AI performance on text-to-SQL has lagged behind. LLM scores on the BIRD leaderboard improved from just below 70% in 2024 to 82% today. Frontier models like…

Putting Task Expertise into RL Achieves State-of-the-Art Performance on Text-to-SQL - Thinking Machines Lab

saved by

related reading