[1909.02060] Distributionally Robust Language Modeling

Abstract:Language models are generally trained on data spanning a wide range of topics (e.g., news, reviews, fiction), but they might be applied to an a priori unknown target distribution (e.g., restaurant reviews). In this paper, we first show that training on text outside the test distribution can degrade test performance when using standard maximum likelihood (MLE) training. To remedy this without the knowledge of the test distribution, we propose an approach which trains a model that performs well over a wide range of potential test distributions. In particular, we derive a new distributionally robust optimization (DRO) procedure which minimizes the loss of the model over the worst-case mixture of topics with sufficient overlap with the training distribution. Our approach, called topic conditional value at risk (topic CVaR), obtains a 5.5 point perplexity reduction over MLE when the language models are trained on a mixture of Yelp reviews and news and tested only on reviews.

Abstract:Language models are generally trained on data spanning a wide range of topics (e.g., news, reviews, fiction), but they might be applied to an a priori unknown target distribution (e.g., restaurant reviews). In this paper, we first show that training on text outside the test distribution can degrade test performance when using standard maximum likelihood (MLE) training. To remedy this without the knowledge of the test distribution, we propose an approach which trains a model that performs well over a wide range of potential test distributions. In particular, we derive a new distributio

saved by

• Akira Yoshiyama

related reading

• [1911.08731] Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalizationarxiv.org
• Large Language Diffusion Modelsarxiv.org
• State of RL for reasoning LLMs | A. Weersaweers.de
• [2305.18290] Direct Preference Optimization: Your Language Model is Secretly a Reward Modelarxiv.org
• Language Modelinglena-voita.github.io
• To Understand Language is to Understand Generalization | Eric Jangevjang.com
• SFT, RL, and On-Policy Distillation Through a Distributional Lens | whnrehiew.github.io
• GitHub - jacobhilton/deep_learning_curriculum: Language model alignment-focused deep learning curriculum · GitHubgithub.com
• Optimizing LLM Test-Time Compute Involves Solving a Meta-RL Problem – Machine Learning Blog | ML@CMU | Carnegie Mellon Universityblog.ml.cmu.edu
• Recent Advances in Language Model Fine-tuningruder.io
• MAI-Thinking-1: Building a Hill-Climbing Machinemicrosoft.ai
• [2506.19733] Breaking Barriers: Do Reinforcement Post Training Gains Transfer To Unseen Domains?arxiv.org