Self-supervised Learning


By Prof. Seungchul Lee
http://iailab.kaist.ac.kr/
Industrial AI Lab at KAIST

Table of Contents

1. Supervised Learning and Transfer Learning

Supervised pretraining on large labeled datasets has led to successful transfer learning

  • ImageNet

  • Pretrain for fine-grained image classification of 1000 classes



  • Use feature representations for downstream tasks, e.g., object detection, image segmentation, and action recognition



But supervised pretraining comes at a cost …

  • Time-consuming and expensive to label datasets for new tasks
  • Domain expertise needed for specialized tasks
    • Radiologists to label medical images
    • Native speakers or language specialists for labeling text in different languages
  • To relieve the burden of labelling,
    • Semi-supervised learning
    • Weakly-supervised learning
    • Unsupervised learning


Self-supervised learning

  • Self-supervised learning (SSL): supervise using labels generated from the data without any manual or weak label sources
    • Sub-class of unsupervised learning
  • Idea: Hide or modify part of the input. Ask model to recover input or classify what changed
    • Self-supervised task referred to as the pretext task can be formulated using only unlabeled data
    • The features obtained from pretext tasks are transferred to downstream tasks like classification, object detection, and segmentation



Pretext Tasks

  • Solving the pretext tasks allow the model to learn good features.

  • We can automatically generate labels for the pretext tasks.