Andrej Karpathy on X: "Finally had a chance to listen through this pod with Sutton, which was interesting and amusing. As background, Sutton's "The Bitter Lesson" has become a bit of biblical text in frontier LLM circles. Researchers routinely talk about and ask whether this or that approach or idea" / X

To view keyboard shortcuts, press question mark View keyboard shortcuts Messages Home 2 Notifications Messages SuperGrok Premium Lists Bookmarks Profile More Post aaron @aarnphm_ Thread Reply See new posts Conversation Andrej Karpathy @karpathy Finally had a chance to listen through this pod with Sutton, which was interesting and amusing. As background, Sutton's "The Bitter Lesson" has become a bit of biblical text in frontier LLM circles. Researchers routinely talk about and ask whether this or that approach or idea is sufficiently "bitter lesson pilled" (meaning arranged so that it benefits from added computation for free) as a proxy for whether it's going to work or worth even pursuing. The underlying assumption being that LLMs are of course highly "bitter lesson pilled" indeed, just look at LLM scaling laws where if you put compute on the x-axis, number go up and to the right. So it's amusing to see that Sutton, the author of the post, is not so sure that LLMs are "bitter lesson p

Andrej Karpathy @karpathy Finally had a chance to listen through this pod with Sutton, which was interesting and amusing. As background, Sutton's "The Bitter Lesson" has become a bit of biblical text in frontier LLM circles. Researchers routinely talk about and ask whether this or that approach or idea is sufficiently "bitter lesson pilled" (meaning arranged so that it benefits from added computation for free) as a proxy for whether it's going to work or worth even pursuing. The underlying assumption being that LLMs are of course highly "bitter lesson pilled" indeed, just look at LLM scaling l

related reading

• Animals vs Ghosts – karpathykarpathy.bearblog.dev
• The bitter lesson of LLM evalsparsed.com
• Does the Bitter Lesson Have Limits?dbreunig.com
• The Bitter Lesson is Misunderstoodobviouslywrong.substack.com
• Richard Sutton – Father of RL thinks LLMs are a dead endyoutube.com
• Andrej Karpathy — AGI is still a decade awaydwarkesh.com
• Richard Sutton – Father of RL thinks LLMs are a dead enddwarkesh.com
• Richard Sutton – Father of RL thinks LLMs are a dead enddwarkesh.com
• The Bitter Lesson is coming for Tokenization – ⛰️ lucalplucalp.dev
• Dwarkesh Patel on X: "The most interesting part for me is where @karpathy describes why LLMs aren't able to learn like humans. As you would expect, he comes up with a wonderfully evocative phrase to describe RL: “sucking supervision bits thx.com
• You can’t imitation-learn how to continual-learn — LessWronglesswrong.com
• Deep Dive into LLMs like ChatGPTyoutube.com