Achilli, Ventura, Silvestri, Pham, Raya, Krotov, Lucibello & Ambrogioni retarget the score-Jacobian spectral-gap estimator from Ventura et al. (2025) -- built for the true (population) score -- to the empirical score of real trained DDPMs, finding new spectral gaps appear that only a theory of the empirical score, not the true score, predicts [achilli-etal-2024-losing-dimensions-geometric-memorization-diffusion] Across real DDPMs (PixelCNN++-backbone U-Nets, 24.5M-61.7M parameters) trained from scratch on MNIST, Fashion-MNIST, CIFAR-10, CelebA-HQ and LSUN-Church at 38 dataset-size splits per dataset, the spectral-gap-estimated local intrinsic dimension declines smoothly across a training-set-size window of roughly 1000-10000 samples (dataset-specific critical points A/B, e.g. CIFAR-10 A=2000/B=16000 of 50000 total) before collapsing toward zero below it [achilli-etal-2024-losing-dimensions-geometric-memorization-diffusion] A secondary participation-ratio diagnostic (effective active-sample count) tracks the same three empirical regimes -- generalization, geometric memorization, and full memorization -- as training-set size shrinks [achilli-etal-2024-losing-dimensions-geometric-memorization-diffusion]

Pham, Raya, Negri, Zaki, Ambrogioni & Krotov map a diffusion model's own energy function E_DM(x_t,t) directly onto a Dense Associative Memory (Hopfield-network) energy E_AM(x), with noise variance playing the role of inverse temperature, and reuse the score-Jacobian SVD/spectral-gap curvature diagnostic from Ventura et al. (2025) and Achilli et al. (2024) to quantify local energy curvature around real DDPM training points [pham-etal-2025-memorization-generalization-diffusion-associative-memory] Across real DDPMs (PixelCNN++-backbone U-Nets) trained from scratch on MNIST, Fashion-MNIST, CIFAR-10 and LSUN-Church at 38 training-set sizes per dataset, nearest-neighbor-distance classification against real and synthetic sample sets identifies memorized, spurious, and generalized regimes whose transition point matches the dimensional-collapse critical points found on the same checkpoints by Achilli et al. (2024) (e.g. CIFAR-10 A=2000/B=16000 of 50000 total) [pham-etal-2025-memorization-generalization-diffusion-associative-memory] A U-Net capacity ablation (widths 64/96/128, 8.9M-109.7M parameters) on CIFAR-10 and LSUN-Church shows the critical training-set size at which the spurious-state fraction peaks scales up with model capacity (CIFAR-10 U-Net64 peak at K=2568 vs. U-Net96 peak at K=5862), directly tying the memorization-geometry transition to model capacity rather than dataset size alone [pham-etal-2025-memorization-generalization-diffusion-associative-memory]

- Recording chain-of-thought hidden-state trajectories during legal reasoning, PCA effective dimensionality d95 (components for 95% variance) falls from 501 to 274 (a 45% collapse) from Llama-3-8B-Instruct to Llama-3.1-70B-Instruct. [anderson-2026-geometry-of-thought] - The paper frames legal reasoning as "Crystallization" - a global-dimensionality collapse, distinct from the discrete-cluster ("Lattice") pattern found in code reasoning. [anderson-2026-geometry-of-thought] - Observational only (a covariance-spectrum statistic on two real checkpoints); no causal intervention. [anderson-2026-geometry-of-thought]

- Across model families, attention-block outputs occupy an intrinsic dimension of only ~60% of the ambient space (directions for 99% variance), versus ~90% for MLP outputs and the residual stream, traced to the attention output projection W^O. [wang-etal-2025-attention-low-rank-subspaces] - The dead-feature rate of SAEs tracks activation intrinsic dimension layer-by-layer: lower-dimensional activations yield more dead features. [wang-etal-2025-attention-low-rank-subspaces] - Active Subspace Initialization (constraining SAE weights to the activations' top singular vectors) cuts dead features from 87% to below 1% in 1M-feature attention-output SAEs while lowering reconstruction loss. [wang-etal-2025-attention-low-rank-subspaces] - Measured on GPT-2, Llama-3.1-8B, Gemma-2-9B, Qwen3-8B/4B and Pythia-160M/2.8B. [wang-etal-2025-attention-low-rank-subspaces]

- Liu et al. introduce FlowLens, a stable PCA-based tool for real residual-stream geometry analysis, and apply it to Llama-3.2-1B (and its Llama-3.2-1B-Instruct/SFT/VCL fine-tuned variants trained on the Tulu-3 recipe, with a broader comparison set including Llama-3.1-8B-Instruct and Llama-2-7B-chat), showing that as the proportion of safety-refusal examples in fine-tuning data rises from 0% to 40%, mid-layer residual-stream variance concentrates increasingly onto a few principal components. [liu-etal-2026-flowlens-variance-concentration-false-refusal] - This measured variance concentration (the paper reports growing variance along, and falling cosine alignment of, a top principal direction -- alignment score 1.0 to 0.85 at 40% safety data in Table 2 -- and describes it as reduced representational smoothness) tracks a concrete behavioral cost: the false-refusal rate (declining benign queries) rises from 63% to 84% (Table 2) over the same safety-data-proportion sweep, establishing a direct link between the geometric collapse and the behavioral failure mode. [liu-etal-2026-flowlens-variance-concentration-false-refusal] - Their own proposed regularizer (Variance Concentration Loss, penalizing excessive mid-layer variance concentration) reduces false refusals by over 35 percentage points while preserving MMLU/GSM8K performance -- but the map-relevant finding is the measured variance-concentration/false-refusal coupling in the real fine-tuned baseline models, not the paper's own fix. [liu-etal-2026-flowlens-variance-concentration-false-refusal]

- Two spectral statistics of last-token representations - RankMe (von Neumann effective rank) and alpha-ReQ (eigenspectrum power-law exponent) - trace a consistent three-phase trajectory across OLMo and Pythia: warmup collapse, entropy-seeking expansion, then compression-seeking consolidation. [li-etal-2025-tracing-representation-geometry] - The entropy-seeking phase tracks peaking short-context n-gram memorization; the compression-seeking phase coincides with a grokking-like rise in downstream (TriviaQA/SciQ) accuracy. Across Pythia 1B-12B and OLMo2-1B, SciQ accuracy correlates with alpha-ReQ at r=0.54-0.84 (p<2.6e-3) and RankMe at r=-0.62 to -0.89. [li-etal-2025-tracing-representation-geometry] - Ablating all but the top-k eigendirections destroys downstream accuracy (Pythia-1B SciQ 0.838 to 0.225 at top-10), so the full spectrum is load-bearing; a proven cross-entropy + bottleneck (d<<|V|, Zipfian) derivation reproduces the phases, and removing either condition or using MSE eliminates compression-seeking. [li-etal-2025-tracing-representation-geometry] - Post-training diverges: SFT/DPO (Tulu-3 pipeline on Llama-3.1-8B) push entropy-seeking while RLVR pushes compression-seeking, with matching diversity-vs-accuracy tradeoffs; the pretraining phases are reported as correlational. [li-etal-2025-tracing-representation-geometry]

- Liu, Chen, Wang, Xu, Lin, Zhang, Zhou, Yang, Wu & Sun define a measurement suite for hidden states, logits, token trajectories, and LoRA updates, and apply it across chains of post-training stages (supervised fine-tuning, preference optimization, safety/refusal tuning, math and code specialization, long chain-of-thought tuning, under controlled stage orderings) on real Qwen2.5-1.5B, TinyLlama-1.1B, Llama-3.2-1B, OLMo-1B, Qwen2.5-7B, and Mistral-7B. [liu-etal-2026-representation-collapse-sequential-post-training] - For one reported model family (Table 4; the identical values are labelled Qwen2.5-1.5B in App. D Table 8), normalized effective rank falls from 0.74 (base) to 0.58 and anisotropy rises from 0.08 to 0.19 across the chain General SFT, Math SFT, Code SFT, Safety SFT, while the target score rises 54.2 to 70.1 and the future-task score falls 71.0 to 60.7. The paper frames this as concentration accumulating over the stage chain, with collapse strongest in middle and late blocks (Fig. 2) and dependent on the objective (Fig. 3); it says its claim is narrower than "post-training always collapses LLMs". [liu-etal-2026-representation-collapse-sequential-post-training] - The paper links this concentration to lower later plasticity (framed in the abstract as a hypothesis, tested with a mixed-effects regression; collapse index vs future score in Table 5 across TinyLlama-1.1B, Llama-3.2-1B, Qwen2.5-1.5B, Qwen2.5-7B), and states that the correlations are not by themselves causal evidence. Weaker out-of-domain generalization and poorer calibration are part of the stated hypothesis; App. B Fig. 9 has calibration/retention panels but the text gives no numerical values for them. The mitigations (mixed-domain replay, feature refresh, representation-diversity regularization, LoRA update decorrelation) are evaluated as a Pareto tradeoff (Fig. 5). [liu-etal-2026-representation-collapse-sequential-post-training]

- SwiGLU feed-forward blocks act as directional quadratic amplifiers, creating massive activations for tokens aligned with a shared "spike direction". [sun-etal-2026-the-spike-the-sparse-and-the-sink] - Pre-norm RMSNorm's bounded-range property (Theorem B.3) collapses these spike tokens' normalized key vectors into a low-dimensional near-constant subspace, and sink heads' query subspaces align with it - explaining attention sinks as geometric alignment, not semantic relevance. [sun-etal-2026-the-spike-the-sparse-and-the-sink] - Causal ablations on a controlled 7B model trained from scratch show spikes and sinks are separable: DynamicTanh eliminates spikes without hurting sinks or perplexity, and long-context-only training sharply reduces the sink ratio. [sun-etal-2026-the-spike-the-sparse-and-the-sink] - Broader analysis applied to pre-norm Llama-family models (approximated here by Llama-2-7B). [sun-etal-2026-the-spike-the-sparse-and-the-sink]

- Across model families, attention-block outputs occupy an intrinsic dimension of only ~60% of the ambient space (directions for 99% variance), versus ~90% for MLP outputs and the residual stream, traced to the attention output projection W^O. [wang-etal-2025-attention-low-rank-subspaces] - The dead-feature rate of SAEs tracks activation intrinsic dimension layer-by-layer: lower-dimensional activations yield more dead features. [wang-etal-2025-attention-low-rank-subspaces] - Active Subspace Initialization (constraining SAE weights to the activations' top singular vectors) cuts dead features from 87% to below 1% in 1M-feature attention-output SAEs while lowering reconstruction loss. [wang-etal-2025-attention-low-rank-subspaces] - Measured on GPT-2, Llama-3.1-8B, Gemma-2-9B, Qwen3-8B/4B and Pythia-160M/2.8B. [wang-etal-2025-attention-low-rank-subspaces]

- Across model families, attention-block outputs occupy an intrinsic dimension of only ~60% of the ambient space (directions for 99% variance), versus ~90% for MLP outputs and the residual stream, traced to the attention output projection W^O. [wang-etal-2025-attention-low-rank-subspaces] - The dead-feature rate of SAEs tracks activation intrinsic dimension layer-by-layer: lower-dimensional activations yield more dead features. [wang-etal-2025-attention-low-rank-subspaces] - Active Subspace Initialization (constraining SAE weights to the activations' top singular vectors) cuts dead features from 87% to below 1% in 1M-feature attention-output SAEs while lowering reconstruction loss. [wang-etal-2025-attention-low-rank-subspaces] - Measured on GPT-2, Llama-3.1-8B, Gemma-2-9B, Qwen3-8B/4B and Pythia-160M/2.8B. [wang-etal-2025-attention-low-rank-subspaces]

- Queipo-de-Llano et al. prove (Theorem 1) that massive residual-stream activations necessarily induce representational compression, with tight bounds on entropy reduction and singular-value dominance. [queipodellano-etal-2025-attention-sinks-and-compression-valleys] - Across Pythia 410M and 6.9B training checkpoints (steps 1k-143k), compression valleys, attention sinks and massive activations emerge synchronized around step 1k and persist, at a fixed layer per model. [queipodellano-etal-2025-attention-sinks-and-compression-valleys] - Staged causal ablation of the massive-activation mechanism progressively - and fully, when all stages are ablated - eliminates the compression valley, confirming the causal link; validated further on GPT-OSS-20B and Gemma-7B. [queipodellano-etal-2025-attention-sinks-and-compression-valleys]

Zolfaghari LoRA-fine-tunes five real model families (Pythia-1.4B, Gemma-2-2B/9B, Qwen2.5-7B, Llama-3.1-8B) into honest and deceptive variants and applies a mechanistic geometry suite (effective rank, participation ratio, Fisher Discriminant Ratio, centroid distance, adjacent-layer direction cosine) to the labeled honest-vs-deceptive activation cloud [zolfaghari-2026-multi-model-study-linear-representations-synthetic-deception] Pythia, Llama, and Qwen collapse the deception-direction cloud to near-rank-1 effective rank (1.06-1.07), while Gemma-2 retains a much higher effective rank (60-234) for the same labeled contrast, an architecture-dependent bifurcation [zolfaghari-2026-multi-model-study-linear-representations-synthetic-deception] Linear-probe decodability of the same honest-vs-deceptive contrast is uniformly near-ceiling (AUC >= 0.99) across all five architectures regardless of the rank-collapse split, showing decodability and rank structure are separate properties [zolfaghari-2026-multi-model-study-linear-representations-synthetic-deception]

- Across model families, attention-block outputs occupy an intrinsic dimension of only ~60% of the ambient space (directions for 99% variance), versus ~90% for MLP outputs and the residual stream, traced to the attention output projection W^O. [wang-etal-2025-attention-low-rank-subspaces] - The dead-feature rate of SAEs tracks activation intrinsic dimension layer-by-layer: lower-dimensional activations yield more dead features. [wang-etal-2025-attention-low-rank-subspaces] - Active Subspace Initialization (constraining SAE weights to the activations' top singular vectors) cuts dead features from 87% to below 1% in 1M-feature attention-output SAEs while lowering reconstruction loss. [wang-etal-2025-attention-low-rank-subspaces] - Measured on GPT-2, Llama-3.1-8B, Gemma-2-9B, Qwen3-8B/4B and Pythia-160M/2.8B. [wang-etal-2025-attention-low-rank-subspaces]

- Roger et al. (2026) repurpose Qwen3-0.6B as a probabilistic time-series forecaster (forecasting cast as next-token prediction over 1,024 discretized bins) and, separately, fit a single linear map on the concatenated frozen hidden states of all 28 layers (D = 28,672) of the language-pretrained model for 1,920 WikiText-103 sequences, trained by EM-style nearest-neighbor matching to a bank of 10,000 real time series without paired supervision: the decoded outputs' nearest neighbors span 686 distinct real series (36% of predictions), versus 54 (2.8%) with random weights and 4 (0.2%) with random-token inputs through pretrained weights (Sec. 4.1, Fig. 3). Retrieving a WikiText projection that matches the first half of a real series gives MSE 1.91 vs 2.27 for last-value carry-forward on 500 series, winning in only 37% of cases (Sec. 4.2). [roger-etal-2026-llm-pretraining-generalizable-manifold] - Across four adaptation regimes (full finetuning, IO-only, LoRA-attention, LoRA-attention+IO), language-pretrained initialization converges faster and to lower forecasting error than random initialization (Fig. 2; effective data transfer D_T = 6.8 / 2.3 / 5.2 / 3.4 for full / IO-only / LoRA-attention / LoRA-attention+IO), and per-sample gradients are already aligned from step one (mean pairwise cosine over 32 held-out series) while the randomly initialized model stays near zero until step 64-256 (Fig. 5). The authors interpret this as a reusable pretrained manifold but state they do not isolate it from initialization statistics (Sec. 6). [roger-etal-2026-llm-pretraining-generalizable-manifold] - Tracking the effective rank of hidden-state covariance (Sec. 5.2, Fig. 7): the randomly initialized model starts near-isotropic (~440) and collapses to ~10 within 500 steps, whereas the language-pretrained model declines from ~50 to ~27 on time-series inputs, with middle layers (~8-17) increasing in effective rank; on text inputs its effective rank drops from ~165 to ~25 (85% reduction) as perplexity rises. [roger-etal-2026-llm-pretraining-generalizable-manifold]

- Liu, Chen, Wang, Xu, Lin, Zhang, Zhou, Yang, Wu & Sun define a measurement suite for hidden states, logits, token trajectories, and LoRA updates, and apply it across chains of post-training stages (supervised fine-tuning, preference optimization, safety/refusal tuning, math and code specialization, long chain-of-thought tuning, under controlled stage orderings) on real Qwen2.5-1.5B, TinyLlama-1.1B, Llama-3.2-1B, OLMo-1B, Qwen2.5-7B, and Mistral-7B. [liu-etal-2026-representation-collapse-sequential-post-training] - For one reported model family (Table 4; the identical values are labelled Qwen2.5-1.5B in App. D Table 8), normalized effective rank falls from 0.74 (base) to 0.58 and anisotropy rises from 0.08 to 0.19 across the chain General SFT, Math SFT, Code SFT, Safety SFT, while the target score rises 54.2 to 70.1 and the future-task score falls 71.0 to 60.7. The paper frames this as concentration accumulating over the stage chain, with collapse strongest in middle and late blocks (Fig. 2) and dependent on the objective (Fig. 3); it says its claim is narrower than "post-training always collapses LLMs". [liu-etal-2026-representation-collapse-sequential-post-training] - The paper links this concentration to lower later plasticity (framed in the abstract as a hypothesis, tested with a mixed-effects regression; collapse index vs future score in Table 5 across TinyLlama-1.1B, Llama-3.2-1B, Qwen2.5-1.5B, Qwen2.5-7B), and states that the correlations are not by themselves causal evidence. Weaker out-of-domain generalization and poorer calibration are part of the stated hypothesis; App. B Fig. 9 has calibration/retention panels but the text gives no numerical values for them. The mitigations (mixed-domain replay, feature refresh, representation-diversity regularization, LoRA update decorrelation) are evaluated as a Pareto tradeoff (Fig. 5). [liu-etal-2026-representation-collapse-sequential-post-training]

- Across model families, attention-block outputs occupy an intrinsic dimension of only ~60% of the ambient space (directions for 99% variance), versus ~90% for MLP outputs and the residual stream, traced to the attention output projection W^O. [wang-etal-2025-attention-low-rank-subspaces] - The dead-feature rate of SAEs tracks activation intrinsic dimension layer-by-layer: lower-dimensional activations yield more dead features. [wang-etal-2025-attention-low-rank-subspaces] - Active Subspace Initialization (constraining SAE weights to the activations' top singular vectors) cuts dead features from 87% to below 1% in 1M-feature attention-output SAEs while lowering reconstruction loss. [wang-etal-2025-attention-low-rank-subspaces] - Measured on GPT-2, Llama-3.1-8B, Gemma-2-9B, Qwen3-8B/4B and Pythia-160M/2.8B. [wang-etal-2025-attention-low-rank-subspaces]

- Queipo-de-Llano et al. prove (Theorem 1) that massive residual-stream activations necessarily induce representational compression, with tight bounds on entropy reduction and singular-value dominance. [queipodellano-etal-2025-attention-sinks-and-compression-valleys] - Across Pythia 410M and 6.9B training checkpoints (steps 1k-143k), compression valleys, attention sinks and massive activations emerge synchronized around step 1k and persist, at a fixed layer per model. [queipodellano-etal-2025-attention-sinks-and-compression-valleys] - Staged causal ablation of the massive-activation mechanism progressively - and fully, when all stages are ablated - eliminates the compression valley, confirming the causal link; validated further on GPT-OSS-20B and Gemma-7B. [queipodellano-etal-2025-attention-sinks-and-compression-valleys]

- Two spectral statistics of last-token representations - RankMe (von Neumann effective rank) and alpha-ReQ (eigenspectrum power-law exponent) - trace a consistent three-phase trajectory across OLMo and Pythia: warmup collapse, entropy-seeking expansion, then compression-seeking consolidation. [li-etal-2025-tracing-representation-geometry] - The entropy-seeking phase tracks peaking short-context n-gram memorization; the compression-seeking phase coincides with a grokking-like rise in downstream (TriviaQA/SciQ) accuracy. Across Pythia 1B-12B and OLMo2-1B, SciQ accuracy correlates with alpha-ReQ at r=0.54-0.84 (p<2.6e-3) and RankMe at r=-0.62 to -0.89. [li-etal-2025-tracing-representation-geometry] - Ablating all but the top-k eigendirections destroys downstream accuracy (Pythia-1B SciQ 0.838 to 0.225 at top-10), so the full spectrum is load-bearing; a proven cross-entropy + bottleneck (d<<|V|, Zipfian) derivation reproduces the phases, and removing either condition or using MSE eliminates compression-seeking. [li-etal-2025-tracing-representation-geometry] - Post-training diverges: SFT/DPO (Tulu-3 pipeline on Llama-3.1-8B) push entropy-seeking while RLVR pushes compression-seeking, with matching diversity-vs-accuracy tradeoffs; the pretraining phases are reported as correlational. [li-etal-2025-tracing-representation-geometry]

Zolfaghari LoRA-fine-tunes five real model families (Pythia-1.4B, Gemma-2-2B/9B, Qwen2.5-7B, Llama-3.1-8B) into honest and deceptive variants and applies a mechanistic geometry suite (effective rank, participation ratio, Fisher Discriminant Ratio, centroid distance, adjacent-layer direction cosine) to the labeled honest-vs-deceptive activation cloud [zolfaghari-2026-multi-model-study-linear-representations-synthetic-deception] Pythia, Llama, and Qwen collapse the deception-direction cloud to near-rank-1 effective rank (1.06-1.07), while Gemma-2 retains a much higher effective rank (60-234) for the same labeled contrast, an architecture-dependent bifurcation [zolfaghari-2026-multi-model-study-linear-representations-synthetic-deception] Linear-probe decodability of the same honest-vs-deceptive contrast is uniformly near-ceiling (AUC >= 0.99) across all five architectures regardless of the rank-collapse split, showing decodability and rank structure are separate properties [zolfaghari-2026-multi-model-study-linear-representations-synthetic-deception]

- The embedding covariance eigenspectrum of six CLIP/SigLIP models shows a sharp elbow (threshold ~10^-3.6) separating a semantic signal subspace from a shared noise subspace. [grzywaczewski-etal-2026-clip-covariance-eigenspectrum-noise] - The noise subspace's dimensionality is quantified per model (CLIP ViT-L/14 164/768 = 21%; SigLIP2 ViT-L/16 405/1024 = 40%) and is shared across unrelated classes (>50% overlap, >90% for the largest models). [grzywaczewski-etal-2026-clip-covariance-eigenspectrum-noise] - Projecting the noise subspace away leaves ImageNet zero-shot accuracy essentially unchanged (CLIP ViT-L/14 91.2% to 91.1%) while removing the same number of random directions degrades it (to 89.3%); text-image cosine similarity even improves. [grzywaczewski-etal-2026-clip-covariance-eigenspectrum-noise] - Tested on CLIP ViT-B/16, ViT-B/32, ViT-L/14, SigLIP ViT-B/16, SigLIP2 ViT-B/16 and ViT-L/16. [grzywaczewski-etal-2026-clip-covariance-eigenspectrum-noise]

- The embedding covariance eigenspectrum of six CLIP/SigLIP models shows a sharp elbow (threshold ~10^-3.6) separating a semantic signal subspace from a shared noise subspace. [grzywaczewski-etal-2026-clip-covariance-eigenspectrum-noise] - The noise subspace's dimensionality is quantified per model (CLIP ViT-L/14 164/768 = 21%; SigLIP2 ViT-L/16 405/1024 = 40%) and is shared across unrelated classes (>50% overlap, >90% for the largest models). [grzywaczewski-etal-2026-clip-covariance-eigenspectrum-noise] - Projecting the noise subspace away leaves ImageNet zero-shot accuracy essentially unchanged (CLIP ViT-L/14 91.2% to 91.1%) while removing the same number of random directions degrades it (to 89.3%); text-image cosine similarity even improves. [grzywaczewski-etal-2026-clip-covariance-eigenspectrum-noise] - Tested on CLIP ViT-B/16, ViT-B/32, ViT-L/14, SigLIP ViT-B/16, SigLIP2 ViT-B/16 and ViT-L/16. [grzywaczewski-etal-2026-clip-covariance-eigenspectrum-noise]

- Contrastive self-supervised embeddings (SimCLR, ResNet-50, ImageNet) undergo dimensional collapse: many singular values of the embedding covariance drop to numerical zero, in both the projection head and the backbone. [jing-etal-2022-dimensional-collapse] - Two causes are proven for a linear model under InfoNCE: strong augmentation drives weights toward zero wherever augmentation variance exceeds data variance (Lemma 1), and multilayer implicit regularization under SGD drives low rank even under weak augmentation (Corollary 2). [jing-etal-2022-dimensional-collapse] - The collapse is diagnosed via the log singular-value spectrum of the covariance; DirectCLR (a fixed low-rank projector) recovers accuracy (ImageNet linear-probe Top-1 66.5% full projector, 51.5% none, 62.7% DirectCLR). [jing-etal-2022-dimensional-collapse]

- In the recurrent-depth model Huginn-3.5B, a four-metric shape suite (entropy, effective rank, anisotropy, intrinsic dimension) applied to latent-thought trajectories separates correct from incorrect reasoning. [latent-thinking-optimization-2025-latent-reasoning-encodes-reward-signals] - Correct trajectories show higher entropy, LOWER effective rank, and higher anisotropy/intrinsic dimension than incorrect ones - a richer but more consolidated latent region. [latent-thinking-optimization-2025-latent-reasoning-encodes-reward-signals] - The signal is reliable enough that a learned Latent Reward Model predicts answer correctness from latent geometry alone (no text decoding), and is used as a training-time reward (Latent Thinking Optimization). [latent-thinking-optimization-2025-latent-reasoning-encodes-reward-signals]

- The Invariant Contamination Ratio (ICR) is a Fisher-based signal-to-noise measure that splits diffusion features into an invariant component (shared across noisy views) and a residual, scoring how much residual contaminates the invariant signal at each noise level. [li-etal-2026-evaluating-representation-space-diffusion-models] - ICR is minimized at intermediate noise levels, which coincide with the best linear-probe classification features (e.g. sigma ~0.29 for EDM, t ~0.2 for SiT). [li-etal-2026-evaluating-representation-space-diffusion-models] - ICR co-moves with FID in data-rich training and its rise anticipates the onset of training-data memorization, enabling label-free early stopping. [li-etal-2026-evaluating-representation-space-diffusion-models] - Tested on EDM, SiT-XL/2 and SiT-B/2 over CIFAR-10/100 and ImageNet-64/256 (Stable Diffusion is only cited as context, not used); observational-diagnostic, no causal intervention. [li-etal-2026-evaluating-representation-space-diffusion-models]

- The Invariant Contamination Ratio (ICR) is a Fisher-based signal-to-noise measure that splits diffusion features into an invariant component (shared across noisy views) and a residual, scoring how much residual contaminates the invariant signal at each noise level. [li-etal-2026-evaluating-representation-space-diffusion-models] - ICR is minimized at intermediate noise levels, which coincide with the best linear-probe classification features (e.g. sigma ~0.29 for EDM, t ~0.2 for SiT). [li-etal-2026-evaluating-representation-space-diffusion-models] - ICR co-moves with FID in data-rich training and its rise anticipates the onset of training-data memorization, enabling label-free early stopping. [li-etal-2026-evaluating-representation-space-diffusion-models] - Tested on EDM, SiT-XL/2 and SiT-B/2 over CIFAR-10/100 and ImageNet-64/256 (Stable Diffusion is only cited as context, not used); observational-diagnostic, no causal intervention. [li-etal-2026-evaluating-representation-space-diffusion-models]

A real ResNet-50 is trained from scratch for 400 epochs (Adam, batch size 512) on real CIFAR-100 (and CIFAR-10, TinyImageNet) with representation dimension set to d = C - 1 and unit-ball/unit-sphere feature normalization, under supervised and unsupervised hard-negative contrastive losses (HSCL, HUCL) [jiang-etal-2023-hard-negative-sampling-neural-dimensional-collapse] Under these conditions the trained representations converge to Neural-Collapse (NC) geometry -- class means forming an Equiangular Tight Frame, measured via the zero-sum, unit-norm, and equal-inner-product metrics matching their theoretical minima -- for a range of hardness levels and batch sizes from 64 to 512 [jiang-etal-2023-hard-negative-sampling-neural-dimensional-collapse] Without incorporating hard negatives or feature normalization, Adam optimization of the same losses on the same real ResNet-50/CIFAR-100 setup instead produces Dimensional-Collapse (DC), failing to attain the NC-ETF geometry [jiang-etal-2023-hard-negative-sampling-neural-dimensional-collapse]

- Queipo-de-Llano et al. prove (Theorem 1) that massive residual-stream activations necessarily induce representational compression, with tight bounds on entropy reduction and singular-value dominance. [queipodellano-etal-2025-attention-sinks-and-compression-valleys] - Across Pythia 410M and 6.9B training checkpoints (steps 1k-143k), compression valleys, attention sinks and massive activations emerge synchronized around step 1k and persist, at a fixed layer per model. [queipodellano-etal-2025-attention-sinks-and-compression-valleys] - Staged causal ablation of the massive-activation mechanism progressively - and fully, when all stages are ablated - eliminates the compression valley, confirming the causal link; validated further on GPT-OSS-20B and Gemma-7B. [queipodellano-etal-2025-attention-sinks-and-compression-valleys]

- Andriopoulos, Dong, Adhikari and Ross estimate the intrinsic dimension of last-layer features (ID_H) and of regression targets (ID_Y) across control tasks and synthetic datasets and report that collapsed regression models show ID_H < ID_Y (over-compression, poor generalization) whereas non-collapsed models typically keep ID_H >= ID_Y, identifying over- and under-compressed regimes that depend on data quantity and noise. Sec. 5 further reports that collapsed last-layer features lie on increasingly lower-dimensional nonlinear (curved) manifolds within the n-dimensional linear subspace. Models are small bespoke regression networks trained by the authors: 3-layer MLPs of varying width on MuJoCo and synthetic image-regression tasks, plus a 4-layer CNN encoder with a 3-layer MLP head on Cheetah-run; a generic custom MLP entry is tagged. [andriopoulos-etal-2025-geometric-analysis-of-neural-regression-collapse-via-intrinsic-dimensi]

Numerical Rank and Effective Rank of the last-hidden-layer node-representation matrix decline sharply as depth increases from 2 to 24 layers in real trained GCN and GAT networks, e.g. dropping from an effective rank of about 1084 to 13.6 on Cora as depth grows [zhang-etal-2025-are-we-measuring-oversmoothing-in-gnns-correctly] Across homophilic (Cora, Citeseer, Pubmed), heterophilic (Squirrel, Chameleon, Amazon Ratings), and large-scale (OGB-Arxiv) real benchmark graphs, rank-based metrics correlate with test accuracy far better than Dirichlet energy (e.g. Cora/GCN: NumRank correlation 0.59, Erank 0.97, vs. Dirichlet energy -0.79) [zhang-etal-2025-are-we-measuring-oversmoothing-in-gnns-correctly] Theorem-level result proves numerical rank converges to 1 for a broad family of GNN architectures as depth grows, paired with the empirical measurement (10 independently trained networks per depth/dataset configuration) rather than left purely theoretical [zhang-etal-2025-are-we-measuring-oversmoothing-in-gnns-correctly]

Numerical Rank and Effective Rank of the last-hidden-layer node-representation matrix decline sharply as depth increases from 2 to 24 layers in real trained GCN and GAT networks, e.g. dropping from an effective rank of about 1084 to 13.6 on Cora as depth grows [zhang-etal-2025-are-we-measuring-oversmoothing-in-gnns-correctly] Across homophilic (Cora, Citeseer, Pubmed), heterophilic (Squirrel, Chameleon, Amazon Ratings), and large-scale (OGB-Arxiv) real benchmark graphs, rank-based metrics correlate with test accuracy far better than Dirichlet energy (e.g. Cora/GCN: NumRank correlation 0.59, Erank 0.97, vs. Dirichlet energy -0.79) [zhang-etal-2025-are-we-measuring-oversmoothing-in-gnns-correctly] Theorem-level result proves numerical rank converges to 1 for a broad family of GNN architectures as depth grows, paired with the empirical measurement (10 independently trained networks per depth/dataset configuration) rather than left purely theoretical [zhang-etal-2025-are-we-measuring-oversmoothing-in-gnns-correctly]

- Two spectral statistics of last-token representations - RankMe (von Neumann effective rank) and alpha-ReQ (eigenspectrum power-law exponent) - trace a consistent three-phase trajectory across OLMo and Pythia: warmup collapse, entropy-seeking expansion, then compression-seeking consolidation. [li-etal-2025-tracing-representation-geometry] - The entropy-seeking phase tracks peaking short-context n-gram memorization; the compression-seeking phase coincides with a grokking-like rise in downstream (TriviaQA/SciQ) accuracy. Across Pythia 1B-12B and OLMo2-1B, SciQ accuracy correlates with alpha-ReQ at r=0.54-0.84 (p<2.6e-3) and RankMe at r=-0.62 to -0.89. [li-etal-2025-tracing-representation-geometry] - Ablating all but the top-k eigendirections destroys downstream accuracy (Pythia-1B SciQ 0.838 to 0.225 at top-10), so the full spectrum is load-bearing; a proven cross-entropy + bottleneck (d<<|V|, Zipfian) derivation reproduces the phases, and removing either condition or using MSE eliminates compression-seeking. [li-etal-2025-tracing-representation-geometry] - Post-training diverges: SFT/DPO (Tulu-3 pipeline on Llama-3.1-8B) push entropy-seeking while RLVR pushes compression-seeking, with matching diversity-vs-accuracy tradeoffs; the pretraining phases are reported as correlational. [li-etal-2025-tracing-representation-geometry]

- Liu, Chen, Wang, Xu, Lin, Zhang, Zhou, Yang, Wu & Sun define a measurement suite for hidden states, logits, token trajectories, and LoRA updates, and apply it across chains of post-training stages (supervised fine-tuning, preference optimization, safety/refusal tuning, math and code specialization, long chain-of-thought tuning, under controlled stage orderings) on real Qwen2.5-1.5B, TinyLlama-1.1B, Llama-3.2-1B, OLMo-1B, Qwen2.5-7B, and Mistral-7B. [liu-etal-2026-representation-collapse-sequential-post-training] - For one reported model family (Table 4; the identical values are labelled Qwen2.5-1.5B in App. D Table 8), normalized effective rank falls from 0.74 (base) to 0.58 and anisotropy rises from 0.08 to 0.19 across the chain General SFT, Math SFT, Code SFT, Safety SFT, while the target score rises 54.2 to 70.1 and the future-task score falls 71.0 to 60.7. The paper frames this as concentration accumulating over the stage chain, with collapse strongest in middle and late blocks (Fig. 2) and dependent on the objective (Fig. 3); it says its claim is narrower than "post-training always collapses LLMs". [liu-etal-2026-representation-collapse-sequential-post-training] - The paper links this concentration to lower later plasticity (framed in the abstract as a hypothesis, tested with a mixed-effects regression; collapse index vs future score in Table 5 across TinyLlama-1.1B, Llama-3.2-1B, Qwen2.5-1.5B, Qwen2.5-7B), and states that the correlations are not by themselves causal evidence. Weaker out-of-domain generalization and poorer calibration are part of the stated hypothesis; App. B Fig. 9 has calibration/retention panels but the text gives no numerical values for them. The mitigations (mixed-domain replay, feature refresh, representation-diversity regularization, LoRA update decorrelation) are evaluated as a Pareto tradeoff (Fig. 5). [liu-etal-2026-representation-collapse-sequential-post-training]

- Two spectral statistics of last-token representations - RankMe (von Neumann effective rank) and alpha-ReQ (eigenspectrum power-law exponent) - trace a consistent three-phase trajectory across OLMo and Pythia: warmup collapse, entropy-seeking expansion, then compression-seeking consolidation. [li-etal-2025-tracing-representation-geometry] - The entropy-seeking phase tracks peaking short-context n-gram memorization; the compression-seeking phase coincides with a grokking-like rise in downstream (TriviaQA/SciQ) accuracy. Across Pythia 1B-12B and OLMo2-1B, SciQ accuracy correlates with alpha-ReQ at r=0.54-0.84 (p<2.6e-3) and RankMe at r=-0.62 to -0.89. [li-etal-2025-tracing-representation-geometry] - Ablating all but the top-k eigendirections destroys downstream accuracy (Pythia-1B SciQ 0.838 to 0.225 at top-10), so the full spectrum is load-bearing; a proven cross-entropy + bottleneck (d<<|V|, Zipfian) derivation reproduces the phases, and removing either condition or using MSE eliminates compression-seeking. [li-etal-2025-tracing-representation-geometry] - Post-training diverges: SFT/DPO (Tulu-3 pipeline on Llama-3.1-8B) push entropy-seeking while RLVR pushes compression-seeking, with matching diversity-vs-accuracy tradeoffs; the pretraining phases are reported as correlational. [li-etal-2025-tracing-representation-geometry]

Von Neumann entropy (from the per-layer Gram matrix) and average curvature of the trajectory traced by adjacent-token transition vectors are computed layer-by-layer for real pretrained Wav2Vec2-base, WavLM-base, HuBERT-base, and Data2Vec-audio-base on the real LibriSpeech test-clean corpus [sadok-alameda-pineda-2026-insidessl] WavLM, HuBERT, and Data2Vec sustain high normalized entropy throughout the network (about 0.82 declining only to about 0.75 in final layers), while Wav2Vec2-base instead undergoes an entropy collapse toward near-baseline levels at layer 11, coinciding with a sharp rise in InfoNCE-based invariance loss (average 1.0 to 3.0) across its final two layers [sadok-alameda-pineda-2026-insidessl] Trajectory curvature starts high (about 1.4) in early layers and decreases to stabilize around 1.2 in deeper layers for most models, indicating the embedding manifold "unfolds" toward linear separability with depth; WavLM, HuBERT and UniSpeech show highly correlated curvature trajectories (Pearson correlation above 0.96), while linear probing shows phoneme-classification accuracy peaks at the depth where curvature transitions from the high- to low-curvature regime [sadok-alameda-pineda-2026-insidessl]

Von Neumann entropy (from the per-layer Gram matrix) and average curvature of the trajectory traced by adjacent-token transition vectors are computed layer-by-layer for real pretrained Wav2Vec2-base, WavLM-base, HuBERT-base, and Data2Vec-audio-base on the real LibriSpeech test-clean corpus [sadok-alameda-pineda-2026-insidessl] WavLM, HuBERT, and Data2Vec sustain high normalized entropy throughout the network (about 0.82 declining only to about 0.75 in final layers), while Wav2Vec2-base instead undergoes an entropy collapse toward near-baseline levels at layer 11, coinciding with a sharp rise in InfoNCE-based invariance loss (average 1.0 to 3.0) across its final two layers [sadok-alameda-pineda-2026-insidessl] Trajectory curvature starts high (about 1.4) in early layers and decreases to stabilize around 1.2 in deeper layers for most models, indicating the embedding manifold "unfolds" toward linear separability with depth; WavLM, HuBERT and UniSpeech show highly correlated curvature trajectories (Pearson correlation above 0.96), while linear probing shows phoneme-classification accuracy peaks at the depth where curvature transitions from the high- to low-curvature regime [sadok-alameda-pineda-2026-insidessl]

Von Neumann entropy (from the per-layer Gram matrix) and average curvature of the trajectory traced by adjacent-token transition vectors are computed layer-by-layer for real pretrained Wav2Vec2-base, WavLM-base, HuBERT-base, and Data2Vec-audio-base on the real LibriSpeech test-clean corpus [sadok-alameda-pineda-2026-insidessl] WavLM, HuBERT, and Data2Vec sustain high normalized entropy throughout the network (about 0.82 declining only to about 0.75 in final layers), while Wav2Vec2-base instead undergoes an entropy collapse toward near-baseline levels at layer 11, coinciding with a sharp rise in InfoNCE-based invariance loss (average 1.0 to 3.0) across its final two layers [sadok-alameda-pineda-2026-insidessl] Trajectory curvature starts high (about 1.4) in early layers and decreases to stabilize around 1.2 in deeper layers for most models, indicating the embedding manifold "unfolds" toward linear separability with depth; WavLM, HuBERT and UniSpeech show highly correlated curvature trajectories (Pearson correlation above 0.96), while linear probing shows phoneme-classification accuracy peaks at the depth where curvature transitions from the high- to low-curvature regime [sadok-alameda-pineda-2026-insidessl]

Von Neumann entropy (from the per-layer Gram matrix) and average curvature of the trajectory traced by adjacent-token transition vectors are computed layer-by-layer for real pretrained Wav2Vec2-base, WavLM-base, HuBERT-base, and Data2Vec-audio-base on the real LibriSpeech test-clean corpus [sadok-alameda-pineda-2026-insidessl] WavLM, HuBERT, and Data2Vec sustain high normalized entropy throughout the network (about 0.82 declining only to about 0.75 in final layers), while Wav2Vec2-base instead undergoes an entropy collapse toward near-baseline levels at layer 11, coinciding with a sharp rise in InfoNCE-based invariance loss (average 1.0 to 3.0) across its final two layers [sadok-alameda-pineda-2026-insidessl] Trajectory curvature starts high (about 1.4) in early layers and decreases to stabilize around 1.2 in deeper layers for most models, indicating the embedding manifold "unfolds" toward linear separability with depth; WavLM, HuBERT and UniSpeech show highly correlated curvature trajectories (Pearson correlation above 0.96), while linear probing shows phoneme-classification accuracy peaks at the depth where curvature transitions from the high- to low-curvature regime [sadok-alameda-pineda-2026-insidessl]

- Liu, Chen, Wang, Xu, Lin, Zhang, Zhou, Yang, Wu & Sun define a measurement suite for hidden states, logits, token trajectories, and LoRA updates, and apply it across chains of post-training stages (supervised fine-tuning, preference optimization, safety/refusal tuning, math and code specialization, long chain-of-thought tuning, under controlled stage orderings) on real Qwen2.5-1.5B, TinyLlama-1.1B, Llama-3.2-1B, OLMo-1B, Qwen2.5-7B, and Mistral-7B. [liu-etal-2026-representation-collapse-sequential-post-training] - For one reported model family (Table 4; the identical values are labelled Qwen2.5-1.5B in App. D Table 8), normalized effective rank falls from 0.74 (base) to 0.58 and anisotropy rises from 0.08 to 0.19 across the chain General SFT, Math SFT, Code SFT, Safety SFT, while the target score rises 54.2 to 70.1 and the future-task score falls 71.0 to 60.7. The paper frames this as concentration accumulating over the stage chain, with collapse strongest in middle and late blocks (Fig. 2) and dependent on the objective (Fig. 3); it says its claim is narrower than "post-training always collapses LLMs". [liu-etal-2026-representation-collapse-sequential-post-training] - The paper links this concentration to lower later plasticity (framed in the abstract as a hypothesis, tested with a mixed-effects regression; collapse index vs future score in Table 5 across TinyLlama-1.1B, Llama-3.2-1B, Qwen2.5-1.5B, Qwen2.5-7B), and states that the correlations are not by themselves causal evidence. Weaker out-of-domain generalization and poorer calibration are part of the stated hypothesis; App. B Fig. 9 has calibration/retention panels but the text gives no numerical values for them. The mitigations (mixed-domain replay, feature refresh, representation-diversity regularization, LoRA update decorrelation) are evaluated as a Pareto tradeoff (Fig. 5). [liu-etal-2026-representation-collapse-sequential-post-training]

- Liu, Chen, Wang, Xu, Lin, Zhang, Zhou, Yang, Wu & Sun define a measurement suite for hidden states, logits, token trajectories, and LoRA updates, and apply it across chains of post-training stages (supervised fine-tuning, preference optimization, safety/refusal tuning, math and code specialization, long chain-of-thought tuning, under controlled stage orderings) on real Qwen2.5-1.5B, TinyLlama-1.1B, Llama-3.2-1B, OLMo-1B, Qwen2.5-7B, and Mistral-7B. [liu-etal-2026-representation-collapse-sequential-post-training] - For one reported model family (Table 4; the identical values are labelled Qwen2.5-1.5B in App. D Table 8), normalized effective rank falls from 0.74 (base) to 0.58 and anisotropy rises from 0.08 to 0.19 across the chain General SFT, Math SFT, Code SFT, Safety SFT, while the target score rises 54.2 to 70.1 and the future-task score falls 71.0 to 60.7. The paper frames this as concentration accumulating over the stage chain, with collapse strongest in middle and late blocks (Fig. 2) and dependent on the objective (Fig. 3); it says its claim is narrower than "post-training always collapses LLMs". [liu-etal-2026-representation-collapse-sequential-post-training] - The paper links this concentration to lower later plasticity (framed in the abstract as a hypothesis, tested with a mixed-effects regression; collapse index vs future score in Table 5 across TinyLlama-1.1B, Llama-3.2-1B, Qwen2.5-1.5B, Qwen2.5-7B), and states that the correlations are not by themselves causal evidence. Weaker out-of-domain generalization and poorer calibration are part of the stated hypothesis; App. B Fig. 9 has calibration/retention panels but the text gives no numerical values for them. The mitigations (mixed-domain replay, feature refresh, representation-diversity regularization, LoRA update decorrelation) are evaluated as a Pareto tradeoff (Fig. 5). [liu-etal-2026-representation-collapse-sequential-post-training]

- SwiGLU feed-forward blocks act as directional quadratic amplifiers, creating massive activations for tokens aligned with a shared "spike direction". [sun-etal-2026-the-spike-the-sparse-and-the-sink] - Pre-norm RMSNorm's bounded-range property (Theorem B.3) collapses these spike tokens' normalized key vectors into a low-dimensional near-constant subspace, and sink heads' query subspaces align with it - explaining attention sinks as geometric alignment, not semantic relevance. [sun-etal-2026-the-spike-the-sparse-and-the-sink] - Causal ablations on a controlled 7B model trained from scratch show spikes and sinks are separable: DynamicTanh eliminates spikes without hurting sinks or perplexity, and long-context-only training sharply reduces the sink ratio. [sun-etal-2026-the-spike-the-sparse-and-the-sink] - Broader analysis applied to pre-norm Llama-family models (approximated here by Llama-2-7B). [sun-etal-2026-the-spike-the-sparse-and-the-sink]

Vemula trains real decoder-only transformers from scratch on FineWeb-Edu at 160M and 290M parameters under four matched configurations (vanilla, QK-normalized "suppress", a novel Ledger-Residuals architecture splitting the residual stream into a mutable Deliberation stream and a protected Commitment stream, and a Commitment variant with 6x stronger commit-sparsity penalty), measuring massive activations via excess kurtosis, fixed-dimension ratio, dominant-dimension persistence, and start-token concentration [vemula-2026-massive-activations-architecturally-robust] At 290M parameters and matched validation loss, the vanilla transformer shows a massive-activation channel at dimension 828 (persistence 0.92, fixed-dimension ratio 28.3, start-token concentration 1.01), which QK-normalization essentially removes (ratio 6.8, concentration 0.13), but the Ledger-Residuals commitment channel rebuilds a massive activation at a different dimension (dimension 22, persistence 0.58, ratio 7.0, concentration 3.36) despite architecturally isolating it from the mutable stream [vemula-2026-massive-activations-architecturally-robust] A 6x stronger commit-sparsity penalty on the same Ledger-Residuals architecture makes the re-emerged massive activation worse rather than better (persistence rises to 0.96, concentration to 4.69), and the same qualitative re-emergence pattern replicates at 160M parameters (vanilla kurtosis 81, ratio 34, persistence 54%; ledger commitment channel rebuilds at ratio 13, concentration 3.04) [vemula-2026-massive-activations-architecturally-robust]

- Wang et al. (2026) report that TabPFN-v2, whose tokenizer maps each scalar cell through a 1 x p linear projection, has low-rank hidden states, "especially early in the network": they run SVD on hidden states across the OpenML-CC18 benchmark and report numerical rank, Rank@99% and Rank@95% per layer (Sec. 3, Fig. 1), and truncating every attention/FFN module input to rank r barely changes TabPFN-v2's AUC (0.9177 at r = 192, 0.9143 at r = 50, 0.8985 at r = 20, 0.7674 at r = 5; Table 1). [wang-etal-2026-limix-2m-low-rank-collapse] - Proposition 3.1 proves that with a linear embedding and no positional embedding the embedded input has rank at most 2 (at most H + 1 after multi-head attention), and Appendix B argues theoretically that feature IDs/positional signals cannot add within-feature value degrees of freedom; these two statements are derivations, not measurements on TabPFN-v2. [wang-etal-2026-limix-2m-low-rank-collapse] - Replacing the linear projection with their RBF-expanded cell encoder (RaBEL) raises first-three-layer rank in a matched 2M-parameter comparison (numerical rank 58.41 to 78.62, Rank@99% 13.94 to 25.35, Rank@95% 6.73 to 12.31; Table 5); the resulting LimiX-2M and the 2M baseline are the authors' own models, and the paper reports LimiX-2M outperforming the larger TabPFN-v2 (7.24M) and TabICL (27.10M) on most benchmarks (Tables 3-6). [wang-etal-2026-limix-2m-low-rank-collapse]

- Zhuo et al. (2023) measure the effective rank (erank) of the normalized outputs of the online and target branches for BYOL, SimSiam, SwAV and DINO (ResNet-18 trained with solo-learn on CIFAR-10): the target branch consistently has higher rank than the online branch and the online rank rises during training after warmup (Fig. 1); they find a high degree of eigenspace alignment between the online and target correlation matrices (always above 0.7 for DINO; Fig. 3), and the filter g = sqrt(lambda^p / lambda^z) is roughly monotonically increasing for all four methods, i.e. a low-pass online filter (Fig. 4b). [zhuo-etal-2023-rank-differential-mechanism] - Under their Hypothesis 1 (eigenspace alignment), Theorem 3 proves that a monotonically increasing g (low-pass online filter) gives erank(C_p) < erank(C_z), and Theorem 4 shows gradient descent then raises the online effective rank; a fully symmetric variant (SymSimSiam) avoids complete collapse but suffers more dimensional collapse than SimSiam (Fig. 2c) and has lower linear-probe accuracy (Fig. 2a), which the authors take to show that asymmetry is what alleviates dimensional collapse. [zhuo-etal-2023-rank-differential-mechanism] - New asymmetric variants built from the theory (fixed online low-pass filters g(sigma) = sigma, log sigma, log(1 + sigma), log(1 + sigma^2), and target high-pass filters h(sigma) = sigma^p with -1 <= p < 0) are compared with SimSiam (and, in Appendix C, in the BYOL framework): on CIFAR-10/100 and ImageNet-100 target high-pass filters often outperform SimSiam in linear-probe accuracy at 100 epochs (e.g. 78.17 vs 74.00 on CIFAR-10) while online filters are comparable (Tables 1-2); on ImageNet-1k, SimSiam with a ResNet-50 backbone scores 67.97 (learnable predictor) vs 64.67 (log(1 + sigma) online filter) and 67.73 (sigma^-0.3 target filter) top-1 (Table 3). [zhuo-etal-2023-rank-differential-mechanism]

- Zhuo et al. (2023) measure the effective rank (erank) of the normalized outputs of the online and target branches for BYOL, SimSiam, SwAV and DINO (ResNet-18 trained with solo-learn on CIFAR-10): the target branch consistently has higher rank than the online branch and the online rank rises during training after warmup (Fig. 1); they find a high degree of eigenspace alignment between the online and target correlation matrices (always above 0.7 for DINO; Fig. 3), and the filter g = sqrt(lambda^p / lambda^z) is roughly monotonically increasing for all four methods, i.e. a low-pass online filter (Fig. 4b). [zhuo-etal-2023-rank-differential-mechanism] - Under their Hypothesis 1 (eigenspace alignment), Theorem 3 proves that a monotonically increasing g (low-pass online filter) gives erank(C_p) < erank(C_z), and Theorem 4 shows gradient descent then raises the online effective rank; a fully symmetric variant (SymSimSiam) avoids complete collapse but suffers more dimensional collapse than SimSiam (Fig. 2c) and has lower linear-probe accuracy (Fig. 2a), which the authors take to show that asymmetry is what alleviates dimensional collapse. [zhuo-etal-2023-rank-differential-mechanism] - New asymmetric variants built from the theory (fixed online low-pass filters g(sigma) = sigma, log sigma, log(1 + sigma), log(1 + sigma^2), and target high-pass filters h(sigma) = sigma^p with -1 <= p < 0) are compared with SimSiam (and, in Appendix C, in the BYOL framework): on CIFAR-10/100 and ImageNet-100 target high-pass filters often outperform SimSiam in linear-probe accuracy at 100 epochs (e.g. 78.17 vs 74.00 on CIFAR-10) while online filters are comparable (Tables 1-2); on ImageNet-1k, SimSiam with a ResNet-50 backbone scores 67.97 (learnable predictor) vs 64.67 (log(1 + sigma) online filter) and 67.73 (sigma^-0.3 target filter) top-1 (Table 3). [zhuo-etal-2023-rank-differential-mechanism]

- Zhuo et al. (2023) measure the effective rank (erank) of the normalized outputs of the online and target branches for BYOL, SimSiam, SwAV and DINO (ResNet-18 trained with solo-learn on CIFAR-10): the target branch consistently has higher rank than the online branch and the online rank rises during training after warmup (Fig. 1); they find a high degree of eigenspace alignment between the online and target correlation matrices (always above 0.7 for DINO; Fig. 3), and the filter g = sqrt(lambda^p / lambda^z) is roughly monotonically increasing for all four methods, i.e. a low-pass online filter (Fig. 4b). [zhuo-etal-2023-rank-differential-mechanism] - Under their Hypothesis 1 (eigenspace alignment), Theorem 3 proves that a monotonically increasing g (low-pass online filter) gives erank(C_p) < erank(C_z), and Theorem 4 shows gradient descent then raises the online effective rank; a fully symmetric variant (SymSimSiam) avoids complete collapse but suffers more dimensional collapse than SimSiam (Fig. 2c) and has lower linear-probe accuracy (Fig. 2a), which the authors take to show that asymmetry is what alleviates dimensional collapse. [zhuo-etal-2023-rank-differential-mechanism] - New asymmetric variants built from the theory (fixed online low-pass filters g(sigma) = sigma, log sigma, log(1 + sigma), log(1 + sigma^2), and target high-pass filters h(sigma) = sigma^p with -1 <= p < 0) are compared with SimSiam (and, in Appendix C, in the BYOL framework): on CIFAR-10/100 and ImageNet-100 target high-pass filters often outperform SimSiam in linear-probe accuracy at 100 epochs (e.g. 78.17 vs 74.00 on CIFAR-10) while online filters are comparable (Tables 1-2); on ImageNet-1k, SimSiam with a ResNet-50 backbone scores 67.97 (learnable predictor) vs 64.67 (log(1 + sigma) online filter) and 67.73 (sigma^-0.3 target filter) top-1 (Table 3). [zhuo-etal-2023-rank-differential-mechanism]

- Zhuo et al. (2023) measure the effective rank (erank) of the normalized outputs of the online and target branches for BYOL, SimSiam, SwAV and DINO (ResNet-18 trained with solo-learn on CIFAR-10): the target branch consistently has higher rank than the online branch and the online rank rises during training after warmup (Fig. 1); they find a high degree of eigenspace alignment between the online and target correlation matrices (always above 0.7 for DINO; Fig. 3), and the filter g = sqrt(lambda^p / lambda^z) is roughly monotonically increasing for all four methods, i.e. a low-pass online filter (Fig. 4b). [zhuo-etal-2023-rank-differential-mechanism] - Under their Hypothesis 1 (eigenspace alignment), Theorem 3 proves that a monotonically increasing g (low-pass online filter) gives erank(C_p) < erank(C_z), and Theorem 4 shows gradient descent then raises the online effective rank; a fully symmetric variant (SymSimSiam) avoids complete collapse but suffers more dimensional collapse than SimSiam (Fig. 2c) and has lower linear-probe accuracy (Fig. 2a), which the authors take to show that asymmetry is what alleviates dimensional collapse. [zhuo-etal-2023-rank-differential-mechanism] - New asymmetric variants built from the theory (fixed online low-pass filters g(sigma) = sigma, log sigma, log(1 + sigma), log(1 + sigma^2), and target high-pass filters h(sigma) = sigma^p with -1 <= p < 0) are compared with SimSiam (and, in Appendix C, in the BYOL framework): on CIFAR-10/100 and ImageNet-100 target high-pass filters often outperform SimSiam in linear-probe accuracy at 100 epochs (e.g. 78.17 vs 74.00 on CIFAR-10) while online filters are comparable (Tables 1-2); on ImageNet-1k, SimSiam with a ResNet-50 backbone scores 67.97 (learnable predictor) vs 64.67 (log(1 + sigma) online filter) and 67.73 (sigma^-0.3 target filter) top-1 (Table 3). [zhuo-etal-2023-rank-differential-mechanism]