Self-Supervised Learning

The self-supervised task, also known as pretext task, guides us to a supervised loss function, e.g. by a Proxy task:

• Rotate: train a model to classify which rotation has been aplied, the idea is that this model need to learn a deep representation of the object
• Relative position: choose a patch, give one of its neighbouring patches, let the model find where the patch was (classification)
• jigsaw puzzle: a more difficult task than the previous: shuffle the patches, the model need to find the permutation.
• Colorization: from grayscale to color.

Self-supervision generates features that are good for downstream tasks.

For example in language modelling, we can predict a part of the input as a target.

Contrastive learning §

Learn representation such that similar samples are close to each other, dissimilar one are far.

Used in visio and language tasks, for example face autentification.

Just build a clever loss function:

$$L_{cont}(x_i,x_j,\theta )=1(y_i=y_j)\Vert f(x_i)-f(x_j){\Vert }_2^2+1(y_i\ne y_j)\max (0,ϵ-\Vert f(x_i)-f(x_j){\Vert }_2)$$

if the labels are equal, minimize le $L_2$ distance squared, if they are different, maximize the $L_2$ distance. Since the loss function need to be bounded from below, we introduce the max. Also we are fine if the distance is more than $ϵ$, a positive “tollerance” parameter.

Triplet Loss §

Introduced in 2015 in FaceNet, the task was recognition of the same person at different poses and angles.

Given one anchor input $x$, we select a positive sample $x^{+}$ and a negative one $x^{-}$.

Our objective is to minimize the triplet loss:

$$L_{trip}(x,x^{+},x^{-})=\sum _{x\in X}\max (0,\Vert f(x)-f(x^{+}){\Vert }_2^2-\Vert f(x)-f(x^{-}){\Vert }_2^2+ϵ)$$

BYOL §

It doesn’t use negative example! Two identical networkds, online and target. We feed the same image, but with two different data transformation.

The Loss is the cosine similarity between the networks (rescaled). The weights of the target network are a time average of the weights of the online network:

$$\xi =(1-\tau )\xi +\tau \theta$$

we update online the $\theta$ weights of the online model.

After training one the online model is used, and the projection layer discarded.

Barlow Twins §

Again two identitcal networks but we feed the same image with a different transformation. The goal is to make the empirical cross-correlation matrix as close as possible to the identity.

$$L_{BT}=\sum _i(1-C_{ii}{)}^2+\lambda \sum _i\sum _{j\ne i}{C}_{ij}^2$$

where

$$C_{ij}=\frac{{\sum }_b{z}_i^A{z}_j^B}{\sqrt{{\sum }_b(z_i^A{)}^2}\sqrt{{\sum }_b(z_i^B{)}^2}}$$