flâneur

The Only Important Technology Is The Internet — Kevin Lu

kevinlu.ai · 3,747 words · saved by 17 readers

Although progress in AI is often attributed to landmark papers – such as transformers, RNNs, or diffusion – this ignores the fundamental bottleneck of artificial intelligence: the data. But what does it mean to have good data? If we truly want to advance AI, instead of studying deep learning optimization, we should be studying the internet. The internet is the technology that actually unlocked the scaling for our AI models. Inspired by the rapid progress made by architectural innovations (in 5 years, going from AlexNet to the Transformer), many researchers have sought better architecture priors. People bet on if we could devise a better architecture than the transformer. In truth, there have been better architectures developed since the transformer – but then why is it so hard to “feel” an improvement since GPT-4? Compute-bound. Once upon a time, methods scaled with compute, and we saw that more efficient methods were better. What mattered was packing our data into the models as effici

The Only Important Technology Is The Internet How can we continue to scale large language models? Published July 2025 Although progress in AI is often attributed to landmark papers – such as transformers , RNNs , or diffusion – this ignores the fundamental bottleneck of artificial intelligence: the data. But what does it mean to have good data? If we truly want to advance AI, instead of studying deep learning optimization, we should be studying the internet. The internet is the technology that actually unlocked the scaling for our AI models. Transformers are a distraction Inspired by the rapid

saved by

related reading