Alex Gajewski

16 followers · 12 following · 750 views

on the atlas · 14

highlights · 21

concept learning as a proxy task to obtain self-supervision
Learning Unsupervised Visual Grounding Through Semantic Self-Supervision
Each Wikipedia article is split into disjoint 100-word chunks, to make a total of 21M documents.
[2005.11401] Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
keep the document encoder (and index) fixed, only fine-tuning the query encoder BERT q and the BART generator.
[2005.11401] Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
query's probability (what is used to create the score) is conditioned on the document's probability
OpenAI API
warm- start it with BERT pre-training
[2002.08909] REALM: Retrieval-Augmented Language Model Pre-Training
his results in one index refresh per ap- proximately 500 training steps. To demonstrate the impor- tance of frequent index refreshes, we compare against using a slower refresh rate.
[2002.08909] REALM: Retrieval-Augmented Language Model Pre-Training
REALM outperforms the largest T5-11B model while being 30 times smaller.
[2002.08909] REALM: Retrieval-Augmented Language Model Pre-Training
200k steps on 64 Google Cloud TPUs, with a batch size of 512
[2002.08909] REALM: Retrieval-Augmented Language Model Pre-Training
can be run on a single machine with a 12GB GPU
[2002.08909] REALM: Retrieval-Augmented Language Model Pre-Training
Documents are greedily split into chunks of up to 288 BERT wordpieces, resulting in just over 13 million retrieval candidates
[2002.08909] REALM: Retrieval-Augmented Language Model Pre-Training
The NaturalQuestions dataset ( Kwiatkowski et al. , 2019 ) consists of naturally occurring Google queries and their answers.
[2002.08909] REALM: Retrieval-Augmented Language Model Pre-Training
cold-start problem, we warm-start Embed input and Embed doc using a simple training objective known as the Inverse Cloze Task (ICT) where, given a sentence, the model is trained to retrieve the document where that sen- tence came from
[2002.08909] REALM: Retrieval-Augmented Language Model Pre-Training
trivial retrieval candidate z that is too informative: if the masked sentence x comes from document z ,
[2002.08909] REALM: Retrieval-Augmented Language Model Pre-Training
ignificantly outperforms other masking strategies
[2002.08909] REALM: Retrieval-Augmented Language Model Pre-Training
we mask salient spans such as “ United Kingdom ” or “ July 1969 ”
[2002.08909] REALM: Retrieval-Augmented Language Model Pre-Training
asynchronous MIPS refreshes
[2002.08909] REALM: Retrieval-Augmented Language Model Pre-Training
only used to select the top k documents. We recompute p ( z | x ) and its gradient, using the fresh θ , for these top k documents after retriev- ing them.
[2002.08909] REALM: Retrieval-Augmented Language Model Pre-Training
Thus, we can employ Maximum Inner Prod- uct Search (MIPS) algorithms to find the approximate top k documents, using running time and storage space that scale sub-linearly with the number of documents ( Ram & Gray , 2012 ; Shrivastava & Li , 2014 ; Shen et al. , 2015 ).
[2002.08909] REALM: Retrieval-Augmented Language Model Pre-Training
We approximate this by instead summing over the top k documents with highest probability under p ( z | x ) —this is reasonable if most documents have near zero probability.
[2002.08909] REALM: Retrieval-Augmented Language Model Pre-Training
p ( y j | z, x ) ∝ exp ( w ⊤ j BERT MASK ( j ) ( join BERT ( x, z body )) )
[2002.08909] REALM: Retrieval-Augmented Language Model Pre-Training
Given an input x , we first retrieve possibly helpful documents z from a knowledge corpus Z .
[2002.08909] REALM: Retrieval-Augmented Language Model Pre-Training