Self-supervised Learning

By Prof. Seungchul Lee
Industrial AI Lab at KAIST

Table of Contents

1. Supervised Learning and Transfer Learning

Supervised pretraining on large labeled datasets has led to successful transfer learning

  • ImageNet

  • Pretrain for fine-grained image classification of 1000 classes

  • Use feature representations for downstream tasks, e.g., object detection, image segmentation, and action recognition

But supervised pretraining comes at a cost …

  • Time-consuming and expensive to label datasets for new tasks
  • Domain expertise needed for specialized tasks
    • Radiologists to label medical images
    • Native speakers or language specialists for labeling text in different languages
  • To relieve the burden of labelling,
    • Semi-supervised learning
    • Weakly-supervised learning
    • Unsupervised learning

Self-supervised learning

  • Self-supervised learning (SSL): supervise using labels generated from the data without any manual or weak label sources
    • Sub-class of unsupervised learning
  • Idea: Hide or modify part of the input. Ask model to recover input or classify what changed
    • Self-supervised task referred to as the pretext task can be formulated using only unlabeled data
    • The features obtained from pretext tasks are transferred to downstream tasks like classification, object detection, and segmentation

Pretext Tasks

  • Solving the pretext tasks allow the model to learn good features.

  • We can automatically generate labels for the pretext tasks.