Fernando Silva

0 followers · 272 views

on the atlas — 42

highlights — 697

Later, we'll see other ways to adapt a model for different applications
Transfer Learning
we will only at the simple setting: when the pretrained model is fine-tuned on each of the downstream tasks.
Transfer Learning
to let ELMo know both left and right contexts, the authors had to train two different unidirectional LMs and then concatenate representations of both of them. In BERT, we don't need to do that: one model is enough.
Transfer Learning
Differently, MLMs see the whole text at once, but some tokens are corrupted: that's why BERT is bidirectional.
Transfer Learning
they do not see the future.
Transfer Learning
At each step, the standard left-to-right LMs predict the next token based on the previous ones.
Transfer Learning
BERT has two training objectives, and the most important of them is the Masked Language Modeling (MLM) objective. is With the MLM objective, at step the following happens
Transfer Learning
the model predicts whether the two sentences are consecutive sentences in some text or not.
Transfer Learning
Another special token is [CLS]. In training, it is used for the NSP objective we'll see next. Once a model is trained, it is used for downstream tasks.
Transfer Learning
In training, BERT sees pairs of sentences separated with a special token-separator
Transfer Learning
We know only the left-to-right language modeling objective, but it is applicable only for decoders where each token can use only previous ones (and does not see the future)
Transfer Learning
BERT's model architecture is very simple and you already know how it works: it's just the Transformer's encoder. What is new, is the training objectives and the way BERT is used for downstream tasks
Transfer Learning
hat changes is the input format for each task: look at the illustration below.
Transfer Learning
n the fine-tuning stage, the model architecture stays the same except for the final linear layer.
Transfer Learning
Formally, if y1,…,yn is a training token sequence, then at the timestep t a model predicts a probability distribution
Transfer Learning
The architecture is a 12-layer Transformer decoder
Transfer Learning
GPT is a Transformer-based left-to-right language model.
Transfer Learning
What is important, now during fine-tuning you have to only use task-aware input transformations (i.e. feed the data in a certain way) instead of modifying model architecture.
Transfer Learning
Then, this model is fine-tuned on each of the downstream tasks
Transfer Learning
GPT/BERT act not as a replacement for word embeddings, but as a replacement for task-specific models.
Transfer Learning
researchers specializing in it kept improving the task-specific model architectures.
Transfer Learning
for coreference resolution, one had to use a specific model designed for this task
Transfer Learning
Note that ELMo/CoVe representations were mainly used to replace the embedding layer, and kept task-specific model architectures almost intact.
Transfer Learning
CoVe/ELMo replace word embeddings, but GPT/BERT replace entire models.
Transfer Learning
ELMo uses task-specific weights to combine representations from the three layers. These are scalars that are learned for each downstream task. The resulting vector, the weighted sum of representations from all layers, is used to represent a word.
Transfer Learning
Layers contain different information → combine them
Transfer Learning
By concatenating these forward and backward vectors we construct a word representation that "knows" about both left and right contexts.
Transfer Learning
for each word we combine representations from the corresponding layers from the forward and backward LSTMs
Transfer Learning
In this way, word representations know their characters by construction, and we can represent even those words we've never seen in training
Transfer Learning
To address these problems, the authors represent words as outputs of a character-level network.
Transfer Learning
What is also interesting, is how the authors get initial word representations (which are then fed to the LSTMs).
Transfer Learning
The model is very simple and it consists of the two-layer LSTM language models: forward and backward. The two models are used so that each token could have both contexts: left and right.
Transfer Learning
ELMo uses representations not from NMT model, but from a language model. Just by replacing word embeddings (GloVe) with embeddings from LM they got a huge improvement for several tasks such as question answering, coreference resolution, sentiment analysis, named entity recognition, and others.
Transfer Learning
For downstream tasks, the authors propose to use the concatenation of both Glove (which represent individual tokens) and CoVe (tokens encoded in context) vectors.
Transfer Learning
CoVe vectors are encoder outputs
Transfer Learning
Therefore, encoder output contains information about both left and right contexts of a token.
Transfer Learning
Note that in this NMT model, the encoder is bidirectional
Transfer Learning
the authors train an LSTM translation model with attention
Transfer Learning
Therefore, vector representations from the encoder contain information about a word's context.
Transfer Learning
NMT encoders learn to "understand" the source sentence
Transfer Learning
CoVe train an NMT system and use its encoder.
Transfer Learning
What does matter is that instead of representing individual words, CoVe and ELMo represent words in context.
Transfer Learning
Note that here you still have a task-specific model for each task, and these task-specific models can be quite different. What's changed is the way we encode words before feeding them to these task-specific models.
Transfer Learning
But if we take a vector for the cat from somewhere inside a language model, this won't be any cat anymore!
Transfer Learning
Since LMs read the context, this vector representation for cat will know that this is the cat that I saw, the one who sat on the mat
Transfer Learning
by training a language model we get much more than by training word embeddings: language models process not just individual words, but sentences/paragraphs/etc
Transfer Learning
Inside a model, LMs too build vector representations for each word, but these vectors represent not just words, but words in context.
Transfer Learning
. Note that for this you don't need any kind of labels!
Transfer Learning
Instead of representing individual words, we can learn to represent words along with the context they are used in.
Transfer Learning
from replacing only word embeddings in task-specific models to replacing entire task-specific models
Transfer Learning