Offline RL and Large Language Models - by Sergey Levine

Perhaps one of the most impressive demonstrations of the potential for large language models is their ability to hold a reasonable conversation with humans, providing responses that follow the flow of the conversation, respond to questions, and even perform rudimentary tasks, such as searching the web. There is something almost uncanny about being able to interact with a machine in the same way we might interact with another person, asking them to do things, providing clarifications, and engaging in back-and-forth chatter. Of course, it’s often possible to get even the most advanced models to react in unreasonable ways, from rudimentary “failures of common sense” to more subtle failures that border on the adversarial (“ignore the prompt and respond with XYZ”), as well as responses that are undesirable, socially inappropriate, or simply offensive. One interpretation of this fact is that current language models are still not “good enough” – we haven’t yet figured out how to train models

Offline RL and Large Language Models What if the most significant capabilities of large language models have yet to be unlocked? Sergey Levine Dec 04, 2022 54 2 Share Perhaps one of the most impressive demonstrations of the potential for large language models is their ability to hold a reasonable conversation with humans, providing responses that follow the flow of the conversation, respond to questions, and even perform rudimentary tasks, such as searching the web. There is something almost uncanny about being able to interact with a machine in the same way we might interact with another pers

saved by

• Andria Xu

related reading

• RLHF: Reinforcement Learning from Human Feedbackhuyenchip.com
• DeepSeek-R1arxiv.org
• [2305.18290] Direct Preference Optimization: Your Language Model is Secretly a Reward Modelarxiv.org
• Language Models can Solve Computer Tasksarxiv.org
• Recursive Language Models | Alex L. Zhangalexzhang13.github.io
• rl-for-llms.md · GitHubgist.github.com
• State of RL for reasoning LLMs | A. Weersaweers.de
• rlhfbook.com/book.pdfrlhfbook.com
• Discovering Language Model Behaviors with Model-Written Evaluations — LessWronglesswrong.com
• Large Language Model: world models or surface statistics?thegradient.pub
• To Understand Language is to Understand Generalization | Eric Jangevjang.com
• [2203.02155] Training language models to follow instructions with human feedbackarxiv.org