[2409.19759] Balancing Cost and Effectiveness of Synthetic Data Generation Strategies for LLMs
Abstract:As large language models (LLMs) are applied to more use cases, creating high quality, task-specific datasets for fine-tuning becomes a bottleneck for model improvement. Using high quality human data has been the most common approach to unlock model performance, but is prohibitively expensive in many scenarios. Several alternative methods have also emerged, such as generating synthetic or hybrid data, but the effectiveness of these approaches remain unclear, especially in resource-constrained scenarios and tasks that are not easily verified. To investigate this, we group various synthetic data generation strategies into three representative categories -- Answer Augmentation, Question Rephrase and New Question -- and study the performance of student LLMs trained under various constraints, namely seed instruction set size and query budget. We demonstrate that these strategies are not equally effective across settings. Notably, the optimal data generation strategy depends strongly on the ratio between the available teacher query budget and the size of the seed instruction set. When this ratio is low, generating new answers to existing questions proves most effective, but as this ratio increases, generating new questions becomes optimal. Across all tasks, we find that choice of augmentation method and other design choices matter substantially more in low to mid data regimes than in high data regimes. We provide a practical framework for selecting the appropriate augmentation method across settings, taking into account additional factors such as the scalability of each method, the importance of verifying synthetic data, and the use of different LLMs for synthetic data generation.
Balancing Cost and Effectiveness of Synthetic Data Generation Strategies for LLMs Yung-Chieh Chan∗ George Pu∗ Apaar Shanker Parth Suresh Penn Jenks John Heyer Sam Denton Scale AI arXiv:2409.19759v3 [cs.CL] 30 Oct 2024 {yungchieh.chan, george.pu,…
saved by
related reading
- Autodata: An agentic data scientist to create high quality synthetic dataalphaxiv.org
- What We’ve Learned From A Year of Building with LLMs – Applied LLMsapplied-llms.org
- BeyondWeb: Lessons from Scaling Synthetic Data for Trillion-scale Pretrainingdatologyai.com
- How to Generate and Use Synthetic Data for Finetuningeugeneyan.com
- RL on Incorrect Synthetic Data Scales the Efficiency of LLM Math Reasoning by Eight-Foldarxiv.org
- Patterns for Building LLM-based Systems & Productseugeneyan.com
- GenAI Handbookgenai-handbook.github.io
- Large Language Diffusion Modelsarxiv.org
- Synthetic Pretraining | Vintage Datavintagedata.org
- PostTrainBenchposttrainbench.com
- Things we learned about LLMs in 2024simonwillison.net
- Data Management For Large Language Models: A Surveyarxiv.org