Training & LearningSSL
Self-Supervised Learning
A form of learning where the model creates its own labels from raw data, enabling training on massive unlabeled datasets.
Self-supervised learning is the secret behind modern foundation models. Instead of requiring human-labeled data, the model generates its own supervision signal from the raw data itself — like predicting the next word in a sentence, or filling in a masked patch of an image.
This unlocks the ability to train on internet-scale data. LLMs are trained self-supervised on trillions of tokens of text, learning grammar, facts, and reasoning without a single labeled example.
Key insight: the data contains the signal. You don't need humans to annotate what a word means when the model can learn meaning from context alone.
Self-supervised learning is behind every major advance in NLP (GPT, BERT) and increasingly in computer vision (MAE, DINO) and multimodal models.