BIRD-bench

BIRD-bench

BIRD-bench --> About BIRD --> BIRD ( BI g Bench for La R ge-scale D atabase Grounded Text-to-SQL Evaluation) represents a pioneering, cross-domain dataset that examines the impact of extensive database contents on text-to-SQL parsing. 1) the necessity to incorporate external knowledge into parsers; 2) the need for parsers to reason over 'dirty' values; and 3) the requirment to consider efficiency while executing SQLs. --> BIRD contains over 12,751 unique question-SQL pairs, 95 big databases with a total size of 33.4 GB . It also covers more than 37 professional domains, such as blockchain, hoc

related reading

• Putting Task Expertise into RL Achieves State-of-the-Art Performance on Text-to-SQLthinkingmachines.ai
• PostTrainBenchposttrainbench.com
• [2005.11401] Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasksarxiv.org
• Scaling Activation Oracles to Trillion-Parameter Modelstransluce.org
• Spider: Yale Semantic Parsing and Text-to-SQL Challengeyale-lily.github.io
• TOD-Bertarxiv.org
• BigO(Bench) -- Can LLMs Generate Code with Controlled Time and Space Complexity?alphaxiv.org
• GitHub - openai/mle-bench: MLE-bench is a benchmark for measuring how well AI agents perform at machine learning engineeringgithub.com
• [2603.07146] Fine-Grained Table Retrieval Through the Lens of Complex Queriesarxiv.org
• ProgramBenchprogrambench.com
• benchmarks.bio — Agentic benchmarks on messy, real-world biological databenchmarks.bio
• Databases in 2025: A Year in Review // Blog // Andy Pavlo - Carnegie Mellon Universitycs.cmu.edu