flâneur

Is Synthetic Data the Key to AGI? - by Nabeel S. Qureshi

digitalspirits.substack.com · 1,466 words · saved by 1 readers

One key fact about modern large language models (LLMs) can be summarized as follows: it’s the dataset, stupid. AI model behavior is largely determined by the dataset it’s trained on; other details, such as architecture, are simply a means of delivering computing power to that dataset. Having a clean, high-quality dataset is worth a lot. [1] The centrality of data is reflected in AI business practice. OpenAI recently announced deals with Axel Springer, Elsevier, the Associated Press, and other publishers and mass media companies for their data; the New York Times (NYT) recently sued OpenAI, demanding that its GPTs, trained on NYT data, be shut down. And Apple is offering $50 million plus for data contracts with publishers. At current margins, models benefit more from additional data than they do from additional size. The expansion in size of training corpora has been rapid. The first modern LLM was trained on Wikipedia. GPT-3 was trained on 300 billion tokens (typically words, parts of

This is part 2 in a series on AI. The first part can be found here. LLMs are trained on vast amounts of data, many libraries’ worth. But what if we run out? Image source: Twitter One key fact about modern large language models (LLMs) can be summarized as follows: it’s the dataset, stupid. AI model behavior is largely determined by the dataset it’s trained on; other details, such as architecture, are simply a means of delivering computing power to that dataset. Having a clean, high-quality dataset is worth a lot. [1] The centrality of data is reflected in AI business practice. OpenAI…

related reading