oLMpics-On What Language Model Pre-training Captures | Transactions of the Association for Computational Linguistics | MIT Press

Alon Talmor, Yanai Elazar, Yoav Goldberg, Jonathan Berant; oLMpics-On What Language Model Pre-training Captures. Transactions of the Association for Computational Linguistics 2020; 8 743–758. doi: https://doi.org/10.1162/tacl_a_00342 Download citation file: Recent success of pre-trained language models (LMs) has spurred widespread interest in the language capabilities that they possess. However, efforts to understand whether LM representations are useful for symbolic reasoning tasks have been limited and scattered. In this work, we propose eight reasoning tasks, which conceptually require operations such as comparison, conjunction, and composition. A fundamental challenge is to understand whether the performance of a LM on a task should be attributed to the pre-trained representations or to the process of fine-tuning on the task data. To address this, we propose an evaluation protocol that includes both zero-shot evaluation (no fine-tuning), as well as comparing the learning curve of a

Alon Talmor, Yanai Elazar, Yoav Goldberg, Jonathan Berant; oLMpics-On What Language Model Pre-training Captures. Transactions of the Association for Computational Linguistics 2020; 8 743–758. doi: https://doi.org/10.1162/tacl_a_00342 Download citation file: Recent success of pre-trained language models (LMs) has spurred widespread interest in the language capabilities that they possess. However, efforts to understand whether LM representations are useful for symbolic reasoning tasks have been limited and scattered. In this work, we propose eight reasoning tasks, which conceptually require oper