Self-supervised Learning

Table of Contents

Many images from https://amitness.com/2020/02/illustrated-self-supervised-learning/

1. Supervised Learning and Transfer Learning

Supervised learning has been a cornerstone of modern machine learning, driving breakthroughs in computer vision, natural language processing, speech recognition, and robotics. For over a decade, the prevailing recipe has been simple: collect data, annotate it, and train a deep neural network end-to-end.

Yet this recipe comes with a hidden cost — labels. As models grow deeper and datasets grow larger, the dependency on human-annotated supervision becomes an increasingly serious bottleneck. In this section, we revisit the fundamentals of supervised learning, diagnose its structural limitations, and survey the landscape of label-efficient alternatives that motivate the study of self-supervised learning.

1.1. What is Supervised Learning?

In supervised learning, a parametric model $f_\theta : \mathcal{X} \to \mathcal{Y}$ is optimized on a labeled dataset

$$\mathcal{D} = \{(x_i, y_i)\}_{i=1}^{N}, \quad x_i \in \mathcal{X},\ y_i \in \mathcal{Y}$$

where $x_i$ is an input observation and $y_i$ is its corresponding ground-truth annotation. The standard training objective minimizes an empirical risk:

$$\min_\theta \; \frac{1}{N} \sum_{i=1}^{N} \mathcal{L}\bigl(f_\theta(x_i),\, y_i\bigr) + \lambda \, \Omega(\theta)$$

where $\mathcal{L}$ is a task-specific loss function and $\Omega(\theta)$ is a regularization term.

The key assumption is that labels $y_i$ encode the relevant semantic or physical signal needed to shape the model's internal representations. When this assumption holds and sufficient labeled data is available, supervised learning is remarkably effective.

1.2. Limitations of Supervised Learning

Despite its empirical success, supervised learning carries three structural limitations that become increasingly acute as we push toward more complex, data-hungry models.

1.2.1. High Labeling Cost

Annotating large-scale datasets is labor-intensive and expensive.

The cost compounds in two ways. First, volume: modern deep networks are data-hungry, and performance on benchmarks like ImageNet continues to improve log-linearly with dataset size. Second, annotation granularity: coarse image-level tags are cheap, but fine-grained labels (pixel masks, 3D bounding boxes, keypoints) are expensive — and unfortunately, many high-value tasks require the latter.

1.2.2. Requirement for Domain Expertise

Many real-world annotation tasks cannot be crowdsourced — they demand specialized knowledge that is simultaneously rare and expensive:

• Medical imaging: Delineating tumors, lesions, or organ boundaries in CT/MRI scans requires board-certified radiologists. Inter-annotator agreement is often moderate even among experts (e.g., Dice scores of 0.7-0.85 for brain tumor segmentation).
• Industrial inspection: Identifying micro-cracks in turbine blades or weld defects in structural components requires trained quality engineers with domain-specific knowledge.
• Scientific discovery: Labeling particle collision events in high-energy physics, annotating cell morphologies in fluorescence microscopy, or classifying geological formations in seismic data all require PhD-level expertise.

This creates a fundamental scalability wall: expert time is finite, expensive, and cannot be parallelized indefinitely.

1.2.3. Limited Cross-Task and Cross-Domain Generalization

Even when a labeled dataset exists, a model trained on it is typically bound to a narrow task-domain combination. When deployed on a different domain $\mathcal{D}_T$ or task $\mathcal{T}_T$, performance often degrades sharply due to distribution shift:

$$P_S(x, y) \neq P_T(x, y)$$

Concrete examples in mechanical engineering contexts:

• A fault diagnosis model trained on bearings from Manufacturer A fails on bearings from Manufacturer B (domain shift in vibration spectra).
• A segmentation model trained on daytime urban scenes degrades at night or in rain (covariate shift).
• A defect detection model trained on steel surfaces does not transfer to composite materials.

Every new task-domain pair, in principle, demands a new annotated dataset. This approach is not scalable.

1.3. Toward Label-Efficient Learning

To mitigate the labeling burden, the machine learning community has developed a family of paradigms that reduce — or eliminate — the need for human annotations.

1.3.1. Semi-Supervised Learning

Semi-supervised learning (Semi-SL) assumes access to a small labeled set $\mathcal{D}_L = \{(x_i, y_i)\}_{i=1}^{M}$ and a large unlabeled set $\mathcal{D}_U = \{x_j\}_{j=1}^{N}$, where $M \ll N$.

The core idea is to exploit the geometric or statistical structure of the input distribution $P(x)$ — accessible from $\mathcal{D}_U$ — to regularize or augment the supervised signal from $\mathcal{D}_L$. Classical approaches include:

• Self-training: Train on $\mathcal{D}_L$, generate pseudo-labels for $\mathcal{D}_U$, and retrain iteratively.
• Label propagation: Propagate labels through a $k$-NN graph over all data points.
• Consistency regularization (e.g., MixMatch, FixMatch): Enforce that predictions are invariant to stochastic augmentations.

Semi-SL is powerful when labeled and unlabeled data share the same distribution, but it still requires some labeled examples and can be sensitive to the quality of pseudo-labels.

1.3.2. Weakly-Supervised Learning

Weakly-supervised learning relaxes the annotation requirement by accepting imprecise, incomplete, or indirect supervision:

A classic example is weakly-supervised object detection: instead of expensive bounding box annotations, only image-level tags are provided. Class Activation Mapping (CAM) techniques can localize discriminative regions despite this coarse supervision.

1.3.3. Unsupervised Learning

Unsupervised learning operates entirely without labels, aiming to discover structure in the raw data distribution $P(x)$. Classical methods include clustering ($k$-means, GMM), dimensionality reduction (PCA, t-SNE, UMAP), and generative modeling (VAE, GAN). These can reveal meaningful structure, but learned representations are often not directly suitable for downstream discriminative tasks without additional supervision.

A Note on Terminology

Self-supervised learning is sometimes categorized as a special case of unsupervised learning. In this course, we treat it as a distinct paradigm: it explicitly constructs supervisory signals from the data itself (e.g., predicting one part of the input from another), rather than simply modeling $P(x)$. This distinction matters — self-supervised representations tend to be far more transferable than those learned by classical unsupervised methods.

1.4. Transfer Learning as a Partial Solution

Before self-supervised learning became dominant, transfer learning was the standard strategy for coping with limited labeled data. Understanding its strengths and limitations directly motivates why self-supervised pre-training is so compelling.

1.4.1. The Transfer Learning Paradigm

Transfer learning proceeds in two stages:

1. Pre-training: Train $f_\theta$ on a large source dataset $\mathcal{D}_{\text{pre}}$ with abundant labels (e.g., ImageNet-1K for vision, BooksCorpus for NLP).
2. Fine-tuning: Adapt $f_\theta$ to a target task by optimizing on a smaller labeled dataset $\mathcal{D}_{\text{fine}}$, with $\theta$ initialized from the pre-trained weights.

$$f_{\theta^*} = \arg\min_\theta \sum_{(x_i, y_i) \in \mathcal{D}_{\text{fine}}} \mathcal{L}\bigl(f_\theta(x_i), y_i\bigr), \quad \theta \leftarrow \theta_{\text{pre-trained}}$$

The empirical success of this recipe is striking: a ResNet-50 pre-trained on ImageNet and fine-tuned on a 1,000-sample medical dataset routinely outperforms the same architecture trained from scratch on 100,000 medical samples.

1.4.2. Why Does Transfer Learning Work?

Pre-training on a large, diverse dataset encourages the model to learn general-purpose feature hierarchies:

• Early layers capture low-level statistics: edges, textures, color gradients — features broadly useful across visual tasks.
• Middle layers encode mid-level structures: object parts, shapes, semantic segments.
• Late layers encode task-specific, high-level semantics tied to the source labels.

When fine-tuning, the task-specific layers are re-trained while the general-purpose lower layers are retained or lightly updated. This dramatically reduces the effective sample complexity of learning the target task.

2. Self-Supervised Learning (SSL)

Self-supervised learning is a paradigm in which supervision signals are derived directly from the structure of the data itself, without relying on human-provided labels or external annotation sources. Rather than requiring a labeled dataset $\mathcal{D} = \{(x_i, y_i)\}$, a self-supervised method constructs a surrogate learning problem — called a pretext task — from the unlabeled data $\mathcal{D}_U = \{x_i\}$ alone.

Formally, given an input $x$, a pretext task defines a transformation $\mathcal{T}$ that produces a modified input $\tilde{x}$ and a corresponding pseudo-label $\tilde{y}$, both derived automatically from $x$:

$$(\tilde{x},\, \tilde{y}) = \mathcal{T}(x)$$

The model $f_\theta$ is then trained to solve this surrogate task:

$$\min_\theta \; \frac{1}{N} \sum_{i=1}^{N} \mathcal{L}\bigl(f_\theta(\tilde{x}_i),\, \tilde{y}_i\bigr)$$

The underlying hypothesis is that in order to solve a well-designed pretext task, the model must learn internal representations that capture the semantic and structural regularities of the data — representations that generalize to downstream tasks of interest.

Pretext Tasks

A pretext task is a self-defined prediction problem whose solution does not require human annotation. The design of the pretext task is crucial: it should be neither too easy (trivially solved without learning useful structure) nor too hard (unsolvable without additional supervision). Representative examples include:

• Masked prediction: A portion of the input is hidden, and the model is trained to reconstruct the missing content. In vision, this corresponds to predicting masked image patches (e.g., MAE); in language, to predicting masked tokens (e.g., BERT). The pseudo-label $\tilde{y}$ is the original unmasked content.

• Geometric transformation prediction: The input is subjected to a geometric transformation — rotation, permutation of patches (jigsaw puzzle), or spatial rearrangement — and the model is trained to predict the transformation parameters. For example, in rotation prediction, $\tilde{y} \in \{0°, 90°, 180°, 270°\}$ is determined entirely by the transformation applied.

• Contrastive learning: Two augmented views $x^+_1$ and $x^+_2$ are generated from the same input $x$, while views from different inputs serve as negatives. The model is trained to produce similar representations for positive pairs and dissimilar representations for negative pairs. The pseudo-label is implicitly defined by the data augmentation process, requiring no human annotation.

• Temporal or contextual prediction: In video or sequential data, the model predicts future frames, missing frames, or surrounding context from observed content. The temporal structure of the data provides the supervisory signal.

From Pretext Tasks to Downstream Tasks

The representations learned through pretext task training are not an end in themselves — they serve as a general-purpose initialization for downstream tasks such as classification, object detection, or semantic segmentation. This transfer typically proceeds in one of two ways:

1. Linear probing: The encoder $f_\theta$ is frozen, and a lightweight linear classifier is trained on top of the learned representations using a small labeled dataset. Strong linear probing performance indicates that the representations are linearly separable with respect to the target labels — a stringent test of representation quality.

2. Fine-tuning: The entire network, including the pre-trained encoder, is updated on the labeled downstream dataset. This is analogous to the supervised transfer learning pipeline described in Section 1.4, except that the pre-trained weights were obtained without any labels.

A central empirical finding in the self-supervised learning literature is that, in low-label regimes, self-supervised pre-training followed by fine-tuning frequently matches or surpasses the performance of fully supervised models trained on the same downstream data. In some settings — particularly where unlabeled data is abundant but labeled data is scarce — self-supervised representations have been shown to outperform supervised pre-training even on the pre-training domain itself.

2.1. Pretext Tasks in Self-Supervised Learning

In self-supervised learning, a pretext task is an auxiliary learning objective that enables a model to learn useful representations from unlabeled data. These tasks are designed so that the labels can be automatically derived from the input data itself, eliminating the need for human annotation.

By solving pretext tasks, the model is encouraged to learn generalizable and transferable features that can be applied to downstream tasks such as image classification, object detection, and segmentation.

The following are representative examples of pretext tasks proposed in prior work. For each example, pay close attention to how the labels are automatically constructed from the raw data.

The image below illustrates four canonical pretext tasks applied to the same input image:

• Image completion: the upper portion of the image is hidden, and the model must reconstruct the missing region from the visible lower patch.
• Rotation prediction: the image is rotated by an unknown angle $\theta$, and the model is trained to predict $\theta$ from the transformed input.
• Jigsaw puzzle solving: the image is divided into a grid of patches that are spatially shuffled, and the model must recover the original arrangement. The adjacent unshuffled image shows the correct reference.
• Colorization: the image is converted to grayscale, and the model is trained to predict the original color values for each pixel. The supervisory signal is the original RGB image, which is available at no annotation cost.

In all four cases, the supervisory signal — the masked region, the rotation angle, the correct patch permutation, or the original color values — is derived automatically from the image itself, with no human annotation required.

(1) Context Prediction

In the context prediction task, an image is divided into nine patches arranged in a $3 \times 3$ grid, numbered 1 through 8 surrounding a central patch. The model receives a pair of patches: one is always taken from the center of the image (blue box), and the other is randomly selected from one of the eight surrounding locations (red dashed boxes). The model is trained to predict the relative spatial position of the second patch with respect to the center patch — a classification problem over 8 possible directions.

The figure below illustrates this pipeline. Given an image $x$, a center-neighbor patch pair is extracted. The model must then identify which of the 8 surrounding positions the neighbor patch was sampled from, as indicated by the $3 \times 3$ grid on the right where the blue box marks the center and the red dashed boxes represent the candidate positions.

To avoid trivial solutions based on low-level cues such as chromatic aberration, edge continuity, or texture statistics, the patches are typically spaced with a gap between them, preventing the model from exploiting simple boundary-matching shortcuts