Learning with not Enough Data Part 1: Semi-Supervised Learning | Lil'Log

lilianweng.github.io · 5,541 words · saved by 2 readers

When facing a limited amount of labeled data for supervised learning tasks, four approaches are commonly discussed. Pre-training + fine-tuning: Pre-train a powerful task-agnostic model on a large unsupervised data corpus, e.g. pre-training LMs on free text, or pre-training vision models on unlabelled images via self-supervised learning, and then fine-tune it on the downstream task with a small set of labeled samples. Semi-supervised learning: Learn from the labelled and unlabeled samples together.

Table of Contents What is semi-supervised learning? Notations Hypotheses Consistency Regularization Π-model Temporal ensembling Mean teachers Noisy samples as learning targets Pseudo Labeling Label propagation Self-Training Reducing confirmation bias Pseudo Labeling with Consistency Regularization MixMatch DivideMix FixMatch Combined with Powerful Pre-Training Citation References When facing a limited amount of labeled data for supervised learning tasks, four approaches are commonly discussed. Pre-training + fine-tuning : Pre-train a powerful task-agnostic model on a large unsupervised data co

Learning with not Enough Data Part 1: Semi-Supervised Learning | Lil'Log

saved by

related reading