Learning with not Enough Data Part 1: Semi-Supervised Learning | Lil'Log

When facing a limited amount of labeled data for supervised learning tasks, four approaches are commonly discussed. Pre-training + fine-tuning: Pre-train a powerful task-agnostic model on a large unsupervised data corpus, e.g. pre-training LMs on free text, or pre-training vision models on unlabelled images via self-supervised learning, and then fine-tune it on the downstream task with a small set of labeled samples. Semi-supervised learning: Learn from the labelled and unlabeled samples together.

Table of Contents What is semi-supervised learning? Notations Hypotheses Consistency Regularization Π-model Temporal ensembling Mean teachers Noisy samples as learning targets Pseudo Labeling Label propagation Self-Training Reducing confirmation bias Pseudo Labeling with Consistency Regularization MixMatch DivideMix FixMatch Combined with Powerful Pre-Training Citation References When facing a limited amount of labeled data for supervised learning tasks, four approaches are commonly discussed. Pre-training + fine-tuning : Pre-train a powerful task-agnostic model on a large unsupervised data co

saved by

• Ellen Xu
• Aidan Smith

related reading

• Self-supervised learning: The dark matter of intelligenceai.facebook.com
• Semi-Supervised Learning: Techniques & Examples [2024]v7labs.com
• On-Policy Distillation - Thinking Machines Labthinkingmachines.ai
• Big Self-Supervised Models are Strong Semi-Supervised Learnersarxiv.org
• Learning with not Enough Data Part 2: Active Learninglilianweng.github.io
• What Is Semi-Supervised Learning - MachineLearningMastery.commachinelearningmastery.com
• Self-Supervised Representation Learning | Lil'Loglilianweng.github.io
• An Overview of Deep Semi-Supervised Learningarxiv.org
• Contrastive Representation Learning | Lil'Loglilianweng.github.io
• How can LLM RL Work Despite Information-Theoretic Inefficiencyberen.io
• Self-supervised learning - Wikipediaen.wikipedia.org
• Self-Labeled Techniques for Semi-Supervised Learning: Taxonomy, Software and Empirical Study | Soft Computing and Intelligent Information Systemssci2s.ugr.es