A Stargate for Data

What would a megaproject for data collection look like?

Labs are on a trajectory towards >$100B/year of data spend by 2030. As we begin the trillion-dollar compute project, we need to think about the equivalent civilizational-scale effort for the other core ingredient: data. At the foundation of the scaling revolution is a simple empirical law: deep neural networks improve smoothly, near magically, as you scale two things in proportion — (1) the size of the model and (2) the amount of data you train on. And despite the scaling laws being brutally diminishing, we’ve successfully bitten the bullet of logarithmic scaling with exponentially larger…

related reading

• A Stargate for Data - by willdepue - Will DePuewilldepue.substack.com
• will depue on X: "A Stargate for Data Labs are on a trajectory towards >$100B/year of data spend by 2030. As we begin the trillion-dollar compute project, we need to think about the equivalent civilizational-scale effort for the other core x.com
• The Only Important Technology Is The Internet - Kevin Lukevinlu.ai
• IIIa. Racing to the Trillion-Dollar Cluster - SITUATIONAL AWARENESSsituational-awareness.ai
• Will scaling work?dwarkeshpatel.com
• There Are No New Ideas in AI… Only New Datasetsblog.jxmo.io
• I. From GPT-4 to AGI: Counting the OOMs - SITUATIONAL AWARENESSsituational-awareness.ai
• Data bottlenecks won’t prevent an intelligence explosionnewsletter.forethought.org
• The data black hole at the center of AIdwarkesh.com
• Training Imperativesdan.io
• Sweatshop data is overmechanize.work
• Can AI scaling continue through 2030? | Epoch AIepoch.ai