Self-Supervised Learning and Collapse

In supervised learning we show an image together with a label a person wrote. Answer "car" on a cat photo and you are wrong. A model that gives every image the same answer also loses on most samples. The external answer holds the model in place.

Unlabeled data has no such answer sheet. That does not mean it has no learning signal. You can mask part of an image and have the model recover the original pixels, or transform one image twice and have it recognize that both came from the same image. Temporally adjacent video frames, or the next word of a sentence, also serve as answers. Nobody wrote the answers down; the structure inside the data provides the supervision. This is the basic idea of self-supervised learning (SSL).

SSL methods answer two big questions. First, what to predict? It can be an observed value like pixels or tokens, or the latent representation of another view. Second, what information must be the same across the two views? Object identity should survive crops and color changes, but the exact background color may not need to. A good pretext task fixes what to discard and what to keep through these two choices.

This is also why matching representations directly is attractive. Pixel reconstruction can spend capacity explaining the grain of grass or sensor noise. In representation space, by contrast, there is room to drop details judged unnecessary for downstream tasks. SimCLR, BYOL, DINO, VICReg and I-JEPA differ in implementation but all exploit this possibility.

But when the model also makes the answer, the external fixed point disappears. Call the two views from one image \(x_1,x_2\) and the encoder \(f_\theta\), and suppose we only shrink the distance between the two representations.

\[\mathcal{L}_{\text{match}}=\mathbb{E}\big[\lVert f_\theta(x_1)-f_\theta(x_2)\rVert^2\big]\]

The solution we want puts cats near cats and apart from cars. But looking at this objective alone, there is an easier solution. Set \(f_\theta(x)=c\) regardless of input and the two terms always agree; the loss is 0. The training problem is solved, and not one bit of information about the data remains.

This constant solution is called a trivial solution, and the phenomenon of the learned representation heading toward that solution or a similar low-dimensional state is called representation collapse. That a constant solution exists in the objective, that optimization actually reaches it, and that downstream performance is poor are not the same sentence. From here on we keep the three apart.

Of course, real methods do not use only the distance term above. SimCLR has negative samples, BYOL has stop-gradient, VICReg has a variance term. Some methods remove the trivial solution from the objective; others leave it in place and keep optimization from flowing there. Understanding collapse means asking what information these extra devices preserve.

In this article the name of an SSL algorithm is not the starting point. We look first at what target it builds, and which collapse it prevents at which level.

Two words first. Modern models do not look at an image whole; they cut it into small patches. Each patch maps to one vector, and that vector is called a token. One image becomes tens to hundreds of tokens, and averaging them gives one vector that summarizes the image. Call it the pooled vector. The "level" in the table below says whether a row is a story between images or inside one image.

The third row is renamed. Bardes et al.'s VICReg calls the state where axes move together informational collapse, but this article uses the narrower term covariance redundancy. The fact that the correlation matrix's off-diagonals are zero cannot by itself certify that the information a downstream task needs has survived.

Most names come from the literature. Hua et al. (2021) split the first two apart, pointing out that dimensional collapse is a distinct and often-overlooked state, and Jing et al. (2022) showed it occurs even in contrastive methods. The bottom two rows have names outside SSL as well: transformer rank collapse (Dong et al., 2021) and over-smoothing in deep ViTs. One more phenomenon, slightly different in kind, belongs alongside: the report of artifact tokens, low-information patches recycled as slots that gather global information (Darcet et al., 2023). V-JEPA 2.1's diagnosis, seen later, has exactly this shape.

A caution here. Patches growing alike can happen regardless of the objective. There is a story that stacking attention deeply does this on its own. But do not inflate the evidence: Dong et al.'s theorem is about pure attention with no skip connections and no MLPs, and the paper's own conclusion is rather that "skip connections play a key role in mitigating rank collapse". A real ViT is not the case the theorem covers. Still, there are separate reports that patch representations grow alike in deep supervised ViTs (Gong et al., 2021; DeepViT). So the discussion below can claim only "observed under SSL objectives", not "caused by SSL objectives".

The six rows overlap, and they are not even concepts on the same level. Dimensional collapse (fewer dimensions in use) and covariance redundancy (dimensions saying the same thing) are usually two descriptions of one event. Mode collapse is a failure of the head that picks prototypes, not of the representation itself, so it is not even defined for VICReg or BYOL, which have no such head. Local structure lost is, strictly, not a collapse. Patches can stay plenty distinct while only their tie to position blurs. They still share one table because the places to hang diagnostics are the same, not because they are the same kind of failure.

Two things to remember in practice. Complete collapse tends to reveal itself readily in modern recipes and can be caught quickly with unnormalized per-dimension standard deviations (the measurement pitfalls come in the next section). Partial dimension loss and local structure loss are much quieter. Also, the top four rows look between images while the bottom two look inside one image. The patches within an image can grow alike while per-image pooled vectors stay distinct, so watching only the upper metrics can miss this failure.

The symptom "the dense features are weak" is usually the bottom two rows. Dense features here are representations for tasks that need one vector per position rather than one per image: segmentation, depth estimation, per-frame prediction. Watch classification accuracy alone and this failure stays invisible to the end.

The figure below compares only the typical shapes of the six states. The two warm-tinted panels are the within-image stories (the same two rows highlighted in the table), and the other four are between-image stories. Real high-dimensional representations are more complicated than a 2-D drawing, so treat this as a concept map, not a diagnostic standard.

Separate two questions first. 1) Has the representation collapsed to a trivial or low-dimensional state? 2) Does the information the downstream task needs remain? Every metric below serves question 1), which is why they can be measured during training without labels. Question 2) is different. As the Multiview assumption of the earlier When Does Collapse Occur? section says, "what counts as needed information" is decided by the task, so no answer comes without a task, a probe or explicit assumptions. Try to answer both questions with one metric and the misunderstanding that spread-out means good begins. Noise or a position code alone can push rank up just fine.

Three terms, unpacked in advance. Effective rank counts "how many of the 512 dimensions are actually in use" (normalize the covariance eigenvalue spectrum, summarize it with entropy, exponentiate). The Gram matrix is the full table of how alike the patches within one image are. When patches clump, this table's rank drops. Perplexity counts "how many of the prepared prototypes are effectively in use".

Standard deviation says different things depending on where you measure it. Measured after normalizing vectors to unit length, you see only direction and miss the collapse where whole vectors shrink. Measured without normalization you see length too, but scales differ across methods and comparisons get hard. Keep both, and read how far things have collapsed from the rank side.

The trap in dividing effective rank by 512: when the batch size \(n\) is smaller than the dimension \(d\), the number of eigenvalues that can even be nonzero is capped at \(\min(n-1,d)\). Even a perfectly spread representation tops out at \((n-1)/d<1\). Read against 1 only when \(n \ge d\); otherwise compare only runs with equal batch sizes.

Mean patch cosine is fooled by sign. Imagine patches lying on a single axis with mixed signs: half at \(+v\), half at \(-v\). Every pair's cosine is +1 or −1, so the mean sits near 0. Looks very healthy. Yet the patches all lie on one 1-D line and the Gram rank is on the floor. Already collapsed, and the mean tells you nothing. So the default metric should be Gram rank, and any cosine reading must come together with an upper percentile (in the case above the top 5% is pinned at 1).

Gram rank is no cure-all either. Patches can all differ while their differences are shuffled independently of position: Gram rank high, yet the distinction between "top-left and bottom-right" already gone. Hence two position-aware metrics: the similarity difference between adjacent and distant patches (the contrast gap), and predicting which grid cell a patch came from given only its vector.

The position probe has one big trap, though. A ViT adds positional information at the input stage, so position survives partly for free. Even a thoroughly collapsed model can score high on it. What this metric measures is not dense performance but "does the patch still remember where it was". Do not read absolute values as performance; read differences between matched conditions, and read whole curves, not single points.

Do not fix thresholds in advance. Embedding scales themselves differ by method, so there is no universal "danger below 0.3". It is right to raise alarms on sharp drops or spikes relative to the early stable stretch of training. If it collapses from the very start there is no stable stretch. Use the value measured at random initialization, before training, as the baseline.

The act of measuring must not change training. Never add these to any loss, of course, and do not use them for learning-rate control or best-checkpoint selection either. Compute them without gradients, and do not run them inside the training forward at every step. Run an SVD per step and training speed differs per condition. Finally, use a separate random stream: draining the randomness used for data order and masking breaks reproducibility.

Finally, back to the two questions of the first paragraph. All the metrics above are tools for question 1). A high Gram rank does not make segmentation work. Not having collapsed is a necessary condition of a usable representation, not a sufficient one. Li, Efros, Pathak (2022) faced this gap head-on, reporting that under partial collapse the metrics and the performance do not move in step.

To measure question 2) you need the task-side tools after all: a linear probe, k-NN, and, if dense is the worry, a segmentation probe on frozen features. These three can also disagree; a pretraining can win at linear probing and lose at fine-tuning, a rank reversal. So in practice it is safest to watch the collapse metrics and the task probes together.

The prediction this table speaks of is narrower than the everyday "predicting something". A method lands in that column only when the target sits at an unobserved position, such as a hidden patch or a future step. Within it, what gets predicted splits again. I-JEPA, V-JEPA and data2vec predict a learned representation \(z\); MAE, BEiT and BEST-RQ predict pixels or fixed tokens \(x\). This \(z\)-versus-\(x\) axis is the same axis as the earlier section on fixed versus jointly learned targets, and is independent of the prediction-versus-view-matching axis.

One common confusion: "it has a predictor" does not mean "prediction side". BYOL has a predictor too. In BYOL the predictor is part of the collapse-blocking asymmetry, unrelated to what is being matched. The predictor is not the criterion that splits the pairing.

The canonical global view-matching recipe summarizes one image into one vector and then compares. In that case the loss carries no condition about which position should become what. In a ViT the summary can be the mean or the CLS token. Either way, what the loss directly sees is one vector.

# prediction side
loss = distance( pred[masked_patch], target[masked_patch] )   # patch <-> patch correspondence

# invariance side (two ways to summarize a ViT)
z1 = projector( view1.tokens.mean(patch_dim) )   # average pooling
z1 = projector( view1.cls )                      # or CLS token
loss = distance(z1, z2)                          # either way, no per-position condition

"The loss never compares patches directly, so patches don't get trained" is wrong. CLS or mean, the gradients flow through attention down into the patch tokens. DINO indeed showed attention maps in which object outlines emerge, with no patch-level term. The precise statement is that there is no per-patch target. Good local features can emerge, but that objective alone does not directly specify per-position properties.

DINOv2 and DINOv3 do not fit the table cleanly. DINO itself belongs in the invariance column, but DINOv2 adds a patch-level iBOT term (Zhou et al., 2022) on top, and DINOv3 stacks Gram anchoring on top of that. They should be read as hybrids straddling both columns.

Here it becomes clear why the criterion set earlier could not be "presence of patch correspondence". DenseCL adds a pixel(patch)-level contrastive term to MoCo-v2: it fixes correspondences between feature-map positions and applies negatives on that correspondence. Patch correspondence sits explicitly in the loss. And yet it is on the invariance side. There is no unobserved position created by masking, so no loss lands on blanks. Both views are fully observed images that only went through augmentation, and corresponding positions pull close while other positions drawn as negatives push away. That is all. Patch correspondence can freely enter the invariance side too (DenseCL · VICRegL), and having a correspondence does not automatically make prediction. Conversely, iBOT's patch term fills the blanks masking created, so it has correspondence and sits on the prediction side.

How the views are made is another, half-independent axis. Multi-crop in particular collides head-on with tasks that demand a different answer per position; when you see the divergence of classification improving while dense degrades, suspect here first.

Augmentation and corruption are also worth separating. Augmentation expects task-relevant semantics to be the same in both views, and view matching writes that expectation into the representation. The Multiview assumption is not another name for augmentation but the stronger sufficiency condition that justifies the expectation. Corruption erases part of the information and has the model recover that target. Denoising autoencoders (Vincent et al., 2008) and MAE · BEiT · data2vec · I-JEPA sit in this lineage. Corruption is not free of assumptions either: it needs the assumption that predicting the erased content builds a useful representation, and that the target is recoverable to a degree from context. The two schemes simply fail under different conditions.

This distinction often interlocks with how views are constructed. The prediction side creates unobserved positions: masking in images and audio, CPC's future steps. The view-matching side leans on augmentations and multi-crop. But masking itself does not imply prediction. MSN matches the summary representation of a masked view to that of an unmasked view, without placing a target on each hidden patch. So by this article's criterion it is on the view-matching side. When porting a method to another domain, compare not only the objective but the original view-construction scheme along with it.

The last column shows the range of the statistic each device computes directly. The base forms applied to global representations mostly handle between-image variance or the batch. Extensions that apply the same principle at the patch level (iBOT, VICRegL) are separated out in the next table.

Once more: these five are not a census. There is one more implementation of the same role (SeLa · SwAV's equipartition, the ancestor of the role centering plays), and at least two devices that fit none of the five. wav2vec 2.0's codebook diversity term keeps a learned codebook from using only a few codes (the table in the When Does Collapse Occur? section concerned codebooks that are not learned). The whitening of the W-MSE family explicitly whitens the embeddings.

Nor does one model live in only one row. The second column does not mean a model uses no other parts. DINO tips into a single entry without centering, and in the ablation that replaces the momentum teacher with a copy of the previous iteration's student, ViT-S/16 k-NN top-1 falls from 72.8% to 0.1%. SimSiam, in its separate recipe, trains without EMA. That result cannot be carried over as "EMA is optional inside BYOL too". Remove the same part and the outcome depends on the model's whole combination.

This table's columns are a different axis from the previous section's prediction/invariance. That one points at the mechanism (does loss land on blanks?), this one at the extension unit (did it go down to patches, or stay pooled?). VICRegL and DenseCL are invariance by mechanism but extended to patch units, so they sit in the left column; conversely, LeWorldModel's next-step prediction is prediction by mechanism but not patch-level, so it sits in the right column (distribution matching row).

The two axes are largely independent. Put stop-gradient on pooled and you get BYOL; on patches, I-JEPA. Hang centering–sharpening on pooled and you get DINO; on patches, iBOT. Draw this table once and you can recognize the same part when you meet it in another cell. The distribution-matching cases covered in this article apply to pooled vectors, and the * on DINOv2 and DINOv3 marks hybrids straddling both cells.

The closest comparison is I-JEPA versus BYOL. The collapse-blocking parts are the same and only the matching scheme differs. But reading this as a clean controlled experiment where only the matching scheme differs is an exaggeration. The view construction (masking vs augmentation), the inputs fed to the encoders, the predictor architecture, and which layer the target comes from all differ together. The accurate reading is as an example that "part names get reused independently of the matching scheme".

Per-image pooled vectors spreading nicely does not guarantee that each patch token preserves spatial information well. The studies that tackled this problem added training signal at different places. First gather those places in one table and then unpack them one by one. The reading key is the two right columns. Type points at what unit the device takes, and Reference needed at what it must hold separately to do so.

For the masked-prediction family a fairly concrete mechanism has been reported. When the loss lands only on masked positions, the encoder has no reason to keep local information at the visible positions. Those representations are never scored; they are only intermediate computation the predictor consults. This is V-JEPA 2.1's diagnosis. The spare capacity goes elsewhere, the visible tokens become depots that gather global information, and the structure of their own positions gets erased. It runs in the same direction as the register token observation that some patches take on global information unrelated to their position. What must not be missed here: this phenomenon does not show up as bad news in global metrics. From the pooled vector's seat, global information gathering well is if anything good news.

Mur-Labadia et al.'s V-JEPA 2.1 (2026) therefore widened V-JEPA 2's prediction targets from masked patches to visible ones as well. The loss becomes two terms: \(\mathcal{L}_{\text{predict}}\) on the masked positions and \(\mathcal{L}_{\text{ctx}}\) on the visible ones. Visible positions do not get the same weight, though; the weight is discounted by the minimum spatiotemporal distance to the masked region (\(\lambda_i=\lambda/\sqrt{d_{\min}(i,\mathcal{M})}\)). Patches hugging the mask boundary get scored hardest, readable as a compromise that demands local continuity while leaving distant tokens room to gather global information.

You have to see the numbers side by side to see this section's point. Adding the context loss took ADE20K segmentation from 22.2 to 33.8 mIoU and NYUv2 depth RMSE from 0.682 to 0.474, big gains. But on the same row, ImageNet-1k fell from 82.2 to 72.6 and SSv2 action recognition from 72.8 to 62.5. The term that saved dense cut global. The paper's way out of the trade is not to weaken the term but to multiply the places it hangs. Deep self-supervision merges representations from three intermediate encoder blocks and the output layer through a light MLP into the predictor and hangs both losses at all four points. Global performance then nearly recovers at IN1K 80.8 / SSv2 72.1 while ADE20K climbs further to 38.6. The warm-up that raises \(\lambda\) slowly over 50–100 epochs is a stabilizer laid on top.

DINOv3 hit the same wall from another angle. As training runs longer, classification keeps improving while the quality of the dense features drops. On ViT-7B, ADE20K mIoU peaks early and loses nearly 5 points by 1M iterations, VOC segmentation turns downward from around 200k steps, and the patch cosine similarity maps grow visibly messy. Watch only global metrics and training looks like it is going well to the very end.

Gram anchoring matches not each patch individually but the table of similarities between patches to a teacher. \(\mathcal{L}_{\text{Gram}}=\lVert X_S X_S^\top - X_G X_G^\top\rVert_F^2\), with \(X\) the \(\ell_2\)-normalized patch feature matrix. Because it does not nail down individual features and only holds the relations, the representation keeps the freedom to translate as a whole. Worth noting that it points the opposite way from the other devices in this section. The rest push patch representations to differ from each other, while Gram anchoring protects structure that was already good.

So what serves as the teacher is nearly the whole of this device. DINOv3 switches this term on in the refinement stage after 1M iterations, and starts the Gram teacher from an early checkpoint, its own earlier self, from when the dense properties were still good. In the ablation there is little difference between checkpoints at 100k and 200k steps, but using the 1M-step one actually hurts. The anchor's value hangs not on "how well trained the teacher is" but on "the teacher from when". During refinement it swaps in the current EMA teacher every 10k steps. The effect appears fast (about 3.3 mIoU of VOC recovers within the first 10k steps), and a high-resolution variant that runs the teacher at 2× resolution and pools it down 2×2 bicubic adds +2 mIoU on ADE20K.

Where the previous two hold positions within one view, DenseCL and VICRegL pair positions across the two views. The pairing rule is the two papers' real difference. DenseCL picks, on a 7×7 backbone feature grid, the partner with the largest cosine similarity (\(c_i=\arg\max_j\,\mathrm{sim}(f_i,f'_j)\)), giving no geometric information and letting the representations match themselves up. VICRegL uses both routes: geometric matching that traces crop coordinates back to the same spot on the source image, and semantic matching by nearest neighbors in embedding space, keeping only the top \(\gamma=20\) pairs of each to filter mismatches. On the paired positions DenseCL applies a local contrastive loss and VICRegL applies the three terms of the earlier section as they are.

The three papers left knobs of the same shape. Raise DenseCL's \(\lambda\) from 0.5 to 0.9 and detection gains 0.8 AP while VOC classification falls 4.8 mAP. Lower VICRegL's \(\alpha\) from 0.9 to 0.75 and segmentation gains several points while ImageNet top-1 slips by under 1%p. V-JEPA 2.1's \(\lambda\), too, trades better segmentation for worse action recognition as it rises. A local term's weight generally shows up as an exchange rate between dense and global. Seen that way, what the recent methods do is not pick a good point on that exchange rate but find places that dodge the exchange: distance weighting (where to hang it), deep supervision (at which depth), a separate refinement stage (when) are each such attempts.

That does not make a local loss a necessary condition. DINO showed dense features in which object boundaries emerge without any explicit patch-level term. There is evidence in the opposite direction too. Gong et al. reported that even in supervised ViTs, patch representations grow alike toward deeper layers. In DeiT-Base24's last layer the mean absolute pairwise cosine similarity exceeds 0.7. If it happens without self-supervision, part of this section's phenomenon belongs to the architecture, not the objective (Open Questions). And this paper using the absolute value is not a detail to skim past: average with signs kept, and patches strung out in opposite directions cancel, giving the illusion of "nicely spread".

In the end, the methods differ in where they hang the loss and what they try to preserve in patch representations. Algorithms 8–10 below show by shape what changes at the point where the earlier five devices stop at \((N,d)\). \(P\) follows all the way into the loss, as \((N,P,K)\) · \((N,G,d)\) · \((N,P,P)\). Spelled out: \(N\) is the number of images in the batch, \(P\) the number of patch tokens in one image, \(d\) the embedding dimension, \(K\) the number of prototypes, and \(G\) the number of local-feature pairs matched across the two views. Read them as the bottom three rows of the table above put into code.

# one target per masked position
ps = head(f(x_masked))    # (N, P, K)
pt = head(f_t(x)).detach()
l = -(pt[mk] * ps[mk].log()).sum(-1)
loss = l.mean()   # mk: (N, P) mask

# loss lands on the M masked patches only
# -> computed on (M, K)

# pair patches across views, three terms
i, j = match(z1, z2)   # top-20 pairs
zl1, zl2 = z1[:, i], z2[:, j]
loss = vicreg(zl1, zl2)  # (N, G, d)

# two matchers: geometry via crop
# coords, and embedding nearest neighbors

# match the relations between patches
u = F.normalize(z_s, dim=-1)  # (N,P,d)
v = F.normalize(z_g, dim=-1)  # Gram t.
Gs, Gt = u @ u.mT, v @ v.mT   # (N,P,P)
diff = (Gs - Gt).pow(2)
loss = diff.sum((-1, -2)).mean()

# z_g starts from an early checkpoint,
# swapped for the EMA teacher every 10k steps

All three keep the \(P\) axis into the loss. That is the token-level loss this section talks about. Gram anchoring's \((N,P,P)\) is especially worth setting beside VICReg's \((d,d)\) covariance. One relates axis to axis, measured along the batch; in the other, \(N\) remains only as a slot and everything closes inside one image.

Under what conditions does stop-gradient reach a non-collapsed solution? This is not a claim that stop-gradient alone suffices. SimSiam, too, showed it within a recipe that includes a predictor and normalization, and BYOL · DINO · JEPA each use further, different parts. The linearized analyses and the batch-statistics counterexamples exist, but a single account carrying those results over to full nonlinear models remains open.

Is an isotropic Gaussian really the right goal? The CCA lineage gives isotropy one motivation, but it is not a proof that representations must take that distribution. Follow-up work already tests other choices. Sub-JEPA applies the Gaussian constraint in several random subspaces, Rectified LpJEPA aims at sparse non-negative target distributions, and UR-JEPA at low-dimensional geometric structure. All three are 2026 preprints, so read them as comparisons in progress rather than established conclusions.

Could the cause be the architecture rather than the objective? If the over-smoothing of deep ViTs happens independently of the objective, part of what this article read as an SSL design problem is an architecture problem. Separating the two effects needs the control that swaps only the objective for supervised learning on the same encoder.

Meeting an unfamiliar method, look at the target first. Whether it is a fixed observation or a jointly learned representation decides whether a constant solution exists. Next, find what excludes or avoids that solution: negatives, asymmetry, the balance of the output distribution, variance–covariance, distribution matching all belong here. Third is the level at which that constraint is computed. Between the images of a batch, or between the patches within one image, the preserved structure differs. Last, set the collapse metrics and a real task probe side by side. The former find trivial solutions; only the latter answers whether the needed information remains.

And the Multiview assumption set aside earlier must be reclaimed here. Collapse-prevention devices are not devices that preserve information. What decides what to compress between the input view and the target view is the augmentation or corruption and the task; all the prevention device does is keep that compression from ending in the trivial solution. When the assumption holds, compressing view-specific detail becomes desirable invariance; when it breaks, the representation becomes insufficient for the task without any complete collapse. So the situation where every collapse metric is normal and performance is still bad is not an anomaly. It is the normal outcome this frame predicts.

Self-supervised learning is the problem of deciding, between an input view and a target view, which information to preserve and which to compress. This includes not only two augmented views but also reconstruction's corrupted input → clean target. Collapse-prevention devices are not devices that preserve all information; they are constraints that keep the compression from ending in the trivial solution. When the level at which that constraint is computed differs from the level the downstream task demands, local information can vanish while the global representation looks healthy.

The token-level cases too fall into place when read through these four questions. Per-position training signal improved dense tasks across several studies, and global metrics could not predict that effect. That does not mean a global objective cannot produce good dense features. The safer, reusable conclusion is short: check the target, the device that avoids collapse, the level that device acts at, and a real task probe, each separately.

Analyses of collapse itself

• Shwartz-Ziv, R., LeCun, Y. (2023). To Compress or Not to Compress — Self-Supervised Learning and Information Theory: A Review. organizes SSL as a multiview information bottleneck and discusses when the assumption that preserving only shared information suffices breaks down.
• Hua, T., Wang, W., Xue, Z., Ren, S., Wang, Y., Zhao, H. (2021). On Feature Decorrelation in Self-Supervised Learning. ICCV. split complete and dimensional collapse apart and named them.
• Tian, Y., Chen, X., Ganguli, S. (2021). Understanding Self-Supervised Learning Dynamics without Contrastive Pairs. ICML. the linearized analysis of why BYOL · SimSiam do not collapse.
• Jing, L., Vincent, P., LeCun, Y., Tian, Y. (2022). Understanding Dimensional Collapse in Contrastive Self-supervised Learning. ICLR.
• Li, A. C., Efros, A. A., Pathak, D. (2022). Understanding Collapse in Non-Contrastive Siamese Representation Learning. ECCV. the relation between partial collapse and performance. What the metrics-are-not-a-proxy section leans on.
• Dong, Y., Cordonnier, J.-B., Loukas, A. (2021). Attention is Not All You Need. ICML. rank collapse in pure attention. The paper's own conclusion, though, is that skip connections mitigate it, so a real ViT is not the theorem's subject.
• Gong, C., Wang, D., Li, M., Chandra, V., Liu, Q. (2021). Vision Transformers with Patch Diversification. the report that patch representations grow alike even in supervised ViTs, and a penalty term against it.
• Zhou, D., Kang, B., Jin, X., et al. (2021). DeepViT: Towards Deeper Vision Transformer.
• Darcet, T., Oquab, M., Mairal, J., Bojanowski, P. (2023). Vision Transformers Need Registers. ICLR 2024. the phenomenon of patches being used to gather global information.

Predict pixels, or match representations?

• Vincent, P., Larochelle, H., Bengio, Y., Manzagol, P.-A. (2008). Extracting and Composing Robust Features with Denoising Autoencoders. ICML. the original of the corruption idea, as contrasted with augmentation.
• LeCun, Y. (2022). A Path Towards Autonomous Machine Intelligence. JEPA's original argument that the unpredictable can be discarded in representation space.
• He, K., Chen, X., Xie, S., Li, Y., Dollár, P., Girshick, R. (2022). Masked Autoencoders Are Scalable Vision Learners. CVPR. for ViT-L, linear probe 75.8 / fine-tuning 85.9 (1600 epochs), and "linear probing and fine-tuning results are largely uncorrelated".

negatives

• van den Oord, A., Li, Y., Vinyals, O. (2018). Representation Learning with Contrastive Predictive Coding (CPC / InfoNCE).
• He, K., Fan, H., Wu, Y., Xie, S., Girshick, R. (2020). Momentum Contrast (MoCo). CVPR.
• Chen, T., Kornblith, S., Norouzi, M., Hinton, G. (2020). A Simple Framework for Contrastive Learning of Visual Representations (SimCLR). ICML.
• Wang, T., Isola, P. (2020). Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere. ICML.
• Baevski, A., Zhou, H., Mohamed, A., Auli, M. (2020). wav2vec 2.0. NeurIPS. the already widespread "within one sample" device, drawing negatives from other time steps of the same utterance.

stop-gradient

• Grill, J.-B., Strub, F., Altché, F., Tallec, C., Richemond, P. H., et al. (2020). Bootstrap Your Own Latent (BYOL). NeurIPS.
• Chen, X., He, K. (2021). Exploring Simple Siamese Representation Learning (SimSiam). CVPR.
• Fetterman, A., Albrecht, J. (2020). Understanding Self-Supervised and Contrastive Learning with BYOL. the hypothesis that the predictor's batch norm secretly plays the negatives' role.
• Richemond, P. H., Grill, J.-B., Altché, F., et al. (2020). BYOL Works Even Without Batch Statistics. the rebuttal to that hypothesis.

Methods that shape distributions

• Asano, Y. M., Rupprecht, C., Vedaldi, A. (2020). Self-labelling via Simultaneous Clustering and Representation Learning (SeLa). ICLR. the source of the equipartition constraint.
• Caron, M., Misra, I., Mairal, J., Goyal, P., Bojanowski, P., Joulin, A. (2020). SwAV. NeurIPS. the implementation that moved it to online, soft assignments on minibatches.
• Caron, M., Touvron, H., Misra, I., Jégou, H., Mairal, J., Bojanowski, P., Joulin, A. (2021). Emerging Properties in Self-Supervised Vision Transformers (DINO). ICCV.
• Assran, M., Caron, M., Misra, I., et al. (2022). Masked Siamese Networks (MSN). ECCV.
• Zhou, J., Wei, C., Wang, H., Shen, W., Xie, C., Yuille, A., Kong, T. (2022). iBOT. ICLR.
• Oquab, M., Darcet, T., Moutakanni, T., Vo, H. V., et al. (2024). DINOv2. TMLR.
• Siméoni, O., Vo, H. V., Seitzer, M., et al. (2025). DINOv3. prescribes Gram anchoring for the degradation of patch representations in long training.

Methods that write statistics into the loss

• Zbontar, J., Jing, L., Misra, I., LeCun, Y., Deny, S. (2021). Barlow Twins. ICML.
• Bardes, A., Ponce, J., LeCun, Y. (2022). VICReg. ICLR.
• Bardes, A., Ponce, J., LeCun, Y. (2022). VICRegL. NeurIPS. the precedent applying the same criteria to paired local features.
• Ermolov, A., Siarohin, A., Sangineto, E., Sebe, N. (2021). Whitening for Self-Supervised Representation Learning (W-MSE). ICML.
• Wang, X., Zhang, R., Shen, C., Kong, T., Li, L. (2021). Dense Contrastive Learning (DenseCL). CVPR. the control where adding only the pixel-level term, on the same backbone and schedule, improved dense performance.
• Balestriero, R., LeCun, Y. (2025). LeJEPA. the paper that introduced SIGReg.
• Maes, L., Le Lidec, Q., Scieur, D., LeCun, Y., Balestriero, R. (2026). LeWorldModel. the construction trained with just two terms, a prediction loss and SIGReg.
• Cramér, H., Wold, H. (1936). Some Theorems on Distribution Functions. J. London Math. Soc. 11(4).
• Epps, T. W., Pulley, L. B. (1983). A Test for Normality Based on the Empirical Characteristic Function. Biometrika 70(3). the original paper of the test statistic SIGReg uses on each 1-D projection.
• Hotelling, H. (1936). Relations Between Two Sets of Variates. Biometrika 28(3/4). the original CCA paper, whitening constraint included.
• Andrew, G., Arora, R., Bilmes, J., Livescu, K. (2013). Deep Canonical Correlation Analysis. ICML. the precedent replacing CCA's projections with neural networks.
• Balestriero, R., LeCun, Y. (2022). Contrastive and Non-Contrastive Self-Supervised Learning Recover Global and Local Spectral Embedding Methods. NeurIPS. reduces VICReg, SimCLR and Barlow Twins to corresponding spectral methods.
• Czinner, S. (2025). The 90-year-old idea behind JEPA models: Canonical Correlation Analysis. the post connecting this lineage directly to JEPA and SIGReg. A personal blog's synthesis, not a peer-reviewed result.
• Zhao, K., et al. (2026). Sub-JEPA. · Kuang, Y., et al. (2026). Rectified LpJEPA. · Le, T. M., et al. (2026). UR-JEPA. follow-up preprints each pushing on "is the isotropic Gaussian the right goal" from a different direction.

Methods with fixed targets

• Bao, H., Dong, L., Piao, S., Wei, F. (2022). BEiT. ICLR.
• Chiu, C.-C., Qin, J., Zhang, Y., Yu, J., Wu, Y. (2022). BEST-RQ. ICML.

Methods that predict in representation space

• Baevski, A., Hsu, W.-N., Xu, Q., Babu, A., Gu, J., Auli, M. (2022). data2vec. ICML.
• Assran, M., Duval, Q., Misra, I., et al. (2023). I-JEPA. CVPR.
• Bardes, A., Garrido, Q., Ponce, J., et al. (2024). V-JEPA.
• Assran, M., Bardes, A., Fan, D., et al. (2025). V-JEPA 2.
• Mur-Labadia, L., Muckley, M., Bar, A., et al. (2026). V-JEPA 2.1: Unlocking Dense Features in Video Self-Supervised Learning.