TOD-Bert

The underlying difference of linguistic patterns between general text and task-oriented dialogue makes existing pre-trained language models less useful in practice. In this work, we unify nine human-human and multi-turn task-oriented dialogue datasets for language modeling. To better model dialogue behavior during pre-training, we incorporate user and system tokens into the masked language modeling. We propose a contrastive objective function to simulate the response selection task. Our pre-trained task-oriented dialogue BERT (TOD-BERT) outperforms strong baselines like BERT on four downstream task-oriented dialogue applications, including intention recognition, dialogue state tracking, dialogue act prediction, and response selection. We also show that TOD-BERT has a stronger few-shot ability that can mitigate the data scarcity problem for task-oriented dialogue.

# link_13dcbt6ngb.pdf ## Metadata - PDFFormatVersion=1.5 - IsLinearized=false - IsAcroFormPresent=false - IsXFAPresent=false - IsCollectionPresent=false - IsSignaturesPresent=false - CreationDate=D:20201002004659Z - Creator=LaTeX with hyperref - ModDate=D:20201002004659Z - Custom.PTEX.Fullbanner=This is pdfTeX, Version 3.14159265-2.6-1.40.21 (TeX Live 2020) kpathsea version 6.3.2 - Producer=pdfTeX-1.40.21 - Trapped=False ## Contents ### Page 1 TOD-BERT: Pre-trained Natural Language Understanding for Task-Oriented DialogueChien-Sheng Wu, Steven Hoi, Richard Socher, and Caiming XiongSalesforc

saved by

• Eric Chang
• Christopher Gresla

related reading

• Exploring Transfer Learning with T5: the Text-To-Text Transfer Transformerai.googleblog.com
• RLHF: Reinforcement Learning from Human Feedbackhuyenchip.com
• Generalized Language Models | Lil'Loglilianweng.github.io
• Transfer Learninglena-voita.github.io
• The Illustrated BERT, ELMo, and co. (How NLP Cracked Transfer Learning) – Jay Alammar – Visualizing machine learning one concept at a time.jalammar.github.io
• [2305.18290] Direct Preference Optimization: Your Language Model is Secretly a Reward Modelarxiv.org
• The Illustrated BERT, ELMo, and co. (How NLP Cracked Transfer Learning) – Jay Alammar – Visualizing machine learning one concept at a time.jalammar.github.io
• Recent Advances in Language Model Fine-tuningruder.io
• Offline RL and Large Language Models - by Sergey Levinesergeylevine.substack.com
• 2405.01470arxiv.org
• MosaicBERT: Pretraining BERT from Scratch for $20 | Databricks Blogmosaicml.com
• [2005.11401] Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasksarxiv.org