MATH · IN · MODELS
structures / Linear Structures / Linear Separability

Linear Separability

PROPERTYlabeled-data-propertybeginnerhow it's classified →

Two activation classes are linearly separable if a hyperplane can separate them — the necessary condition for a clean linear feature direction to exist.

Replicationcomputed from the corpus — never hand-assigned
66 papers · no shared authors11 architecture classes · across papers11 domains · across papers65 model families · across papers
Filled = two or more values reported by papers that share no author — replication. Outlined = two or more values, but all from a single study — breadth, not replication. Grey = a single value. Derived from paper authorship and each model's architecture class, domain and family; it updates itself when a paper is added.

Definition

Two sets A,B⊂RdA,B \subset \mathbb{R}^d are linearly separable if there exist w∈Rdw \in \mathbb{R}^d, b∈Rb\in\mathbb{R} such that

w⋅x+b>0  ∀x∈A,w⋅x+b<0  ∀x∈Bw\cdot x+b>0 \ \ \forall x\in A, \qquad w\cdot x+b<0\ \ \forall x\in B

The hyperplane {x:w⋅x+b=0}\{x: w\cdot x+b=0\} is the decision boundary. This is the strict form; for finite sets it is equivalent to conv⁡(A)∩conv⁡(B)=∅\operatorname{conv}(A)\cap\operatorname{conv}(B)=\varnothing. For general (infinite) sets, guaranteeing a strict gap needs a topological condition (e.g. one set compact and the other closed, or positive distance between them); disjoint convex sets alone give only weak (non-strict) separation.

Homogeneous vs. affine. With b≠0b\neq 0 this is affine separability (the boundary is an affine hyperplane x0+w⊥x_0+w^\perp). Strictly linear (homogeneous) separability requires a boundary through the origin, w⋅x=0w\cdot x=0. The ML term “linear classifier” usually allows a bias, but for a mathematical taxonomy the distinction is worth keeping — one can separately track homogeneous-linear-separability, affine-separability, margin-separability, and approximate-linear-decodability.

What it does and does not guarantee. Separability is a property of a labeled sample: it guarantees a perfect linear classifier exists on that sample, but not causality, out-of-sample stability, uniqueness of the direction ww, localization of the concept into one dimension, or independence from confounders. It is necessary — not sufficient — for a clean single Linear Direction.

Intuition

In R2\mathbb{R}^2: can you draw a single straight line with all of class AA on one side and all of class BB on the other? In Rd\mathbb{R}^d: the same question with a flat hyperplane instead of a line.

Properties

  • Margin. The quality of a separation is margin=min⁡x∈A∪B∣w⋅x+b∣∥w∥\text{margin} = \min_{x\in A\cup B} \dfrac{|w\cdot x+b|}{\|w\|} — the smallest distance from any point to the hyperplane. A larger margin means the classes are more robustly, not just technically, separated.

  • Not scale-invariant in the naive sense, but the boundary is. Scaling (w,b)↦(λw,λb)(w,b)\mapsto(\lambda w,\lambda b) for λ>0\lambda>0 leaves the decision boundary {x:w⋅x+b=0}\{x:w\cdot x+b=0\} and the sign of w⋅x+bw\cdot x+b unchanged, but changes the raw margin formula’s numerator and denominator by the same factor λ\lambda, so the margin itself (as defined above) is also scale-invariant.

  • Convex-hull characterization. A,BA,B are linearly separable if and only if the convex hulls of AA and BB are disjoint — this is a consequence of the separating hyperplane theorem for disjoint convex sets. Zhou & Srikumar (2021) operationalize this directly on real contextualized embeddings (BERT, RoBERTa, ELMo): Convex-hull clustering (DirectProbe) finds the fewest convex, single-label regions consistent with a task’s labels without ever fitting a classifier, and shows the resulting cluster count and inter-cluster margin predict classifier accuracy — see the directprobe-convex-clusters Observation. A follow-up (Zhou & Srikumar, arXiv:2106.14282) tracks this same convex-cluster geometry across fine-tuning: fine-tuning consistently widens the minimum inter-cluster margin between labels (the direct geometric driver of improved classification), while a new Spatial similarity (inter-cluster distance correlation) metric shows this widening largely preserves, rather than replaces, the pre-fine-tuning cluster arrangement — see finetuning-preserves-cluster-geometry.

  • A property of a dataset, not a claim about internal computation. Linear separability describes whether a hyperplane exists for the given points; it says nothing about whether a system that produced those points uses that hyperplane’s normal direction in its own computations. Establishing that requires an intervention (e.g. checking that moving a point along ww changes the system’s behavior), not just fitting a classifier.

  • Necessary, not sufficient, for a clean single Linear Direction. Separability only requires some hyperplane to exist; a single well-defined feature direction additionally requires that the same ww (up to scale) continues to work as the “value” of a graded, not just binary, feature — a stronger requirement.

  • Direction-to-vocabulary alignment can be quantitatively imprinted by training data composition. Kim & Lee (2026) probe language identity across all 268 layers of six multilingual LLMs, finding near-universal linear separability (99.8±0.1% average accuracy, emerging with a +76.4±8.2 percentage-point jump in the first transformer block) and introducing Token-Language Alignment — cosine similarity between a probe-learned language direction and every vocabulary/unembedding token embedding. The resulting alignment (Match@Peak) is not uniform across languages sharing a script: Chinese-inclusive-pretrained models reach 16.43% Chinese alignment versus 3.90% for English-centric models (a 4.21x ratio), while Latin-script languages stay uniformly low regardless of pretraining mix — evidence that a language direction’s alignment with the model’s own vocabulary geometry, not just its separability, is a data-dependent structural property. See kim-lee-2026-language-directions-are-linearly-separable-from-layer-1-and-their-vocabulary-alignment-is-structurally-imprinted-by-pretraining-language-mix.

  • Compounds with low intrinsic dimensionality, and can carry real causal weight. Saglam et al. (2025) measure both properties independently across 11 autoregressive models (GPT-2 through Mistral Small 3 24B): topic-cluster separability (hard-margin SVM) rises toward the final layers while the PCA-measured intrinsic dimensionality of the same activations stays low (“often under 10%” of the ambient dimension needed to explain 90% of variance) — separability and low dimensionality are two distinct, mutually reinforcing measurements of the same underlying geometry, not one implying the other. They also show the separating hyperplane’s normal is not just descriptive: adding the centroid-difference direction between chain-of-thought-framed and plain prompts to a hidden state causally induces CoT-style responses, and a probe built on the same final-layer separability substantially improves refusal rates against jailbreak prompts — see topic-subspaces-separability-sharpens-with-depth-and-reasoning-framing.

  • Sparse-autoencoder features causally editable at the patch level reveal a contrastive-vs-self-supervised split: CLIP’s activation geometry encodes abstract, style-invariant concepts absent or fragmented in DINOv2’s more visually granular feature geometry. Stevens, Chao, Berger-Wolf & Su (2025) train SAEs (24K-width) on frozen CLIP ViT-B/16 and DINOv2 ViT-B/14 patch activations, pairing discovered features with real-image exemplars and decoding vectors; editing a patch’s SAE features causally flips its predicted classification/segmentation class on ImageNet-1K and ADE20K, without retraining the ViT or task heads. See sae-features-reveal-clip-encodes-abstract-style-invariant-concepts-while-dinov2-encodes-fragmented-visually-granular-features-and-are-causally-editable.

  • A parameter-free Nearest Class-Mean clustering measure, applied at both the sample level and the semantic-class level, shows self-supervised training drives emergent clustering by semantic class beyond the augmentation-invariance objective it was designed to optimize. Ben-Shaul, Galanti, Shwartz-Ziv, Dekel & LeCun (2023) train VICReg/SimCLR objectives from scratch on custom ResNet and ViT backbones (CIFAR-100, FOOD101), finding NCC accuracy against true semantic classes rises with both training time and network depth, and exceeds NCC accuracy against random label partitions of equal size. See reverse-engineering-self-supervised-learning-shows-ssl-training-drives-emergent-semantic-class-clustering-beyond-its-designed-augmentation-invariance-objective.

  • Linear separability of an abstract concept emerges only above a hidden-dimension threshold, and this exact same emergent structure is what direction-based jailbreaks exploit — making dimension-reduction a defense. Teo, Abdullaev & Nguyen (2025) show via PCA across Qwen0.5B, GPT2-XL, Qwen7B, and Llama2-7B that final-layer emotion clusters become “distinct, separable” only “in models with more than 3,000 dimensions,” with linear-probe accuracy showing “an obvious distinction between models below and above 2,000 hidden dimensions.” Because white-box jailbreaks (ActAdd/Ablation) work by exploiting exactly this emergent linear safety direction, compressing the ambient dimension back down — via a random Fast Johnson-Lindenstrauss projection on attention Q/K matrices, or a learned linear bottleneck autoencoder on the residual stream — restores collapsed refusal/safety scores (e.g. Llama2-7B-Chat 0.03/0.14 under attack to 0.94-0.96 defended) while retaining most benign helpfulness, backed by a Rademacher-complexity bound tying steering-vector sample complexity to D/N\sqrt{D/N}. See teo-etal-2025-blessing-and-curse-of-dimensionality-in-safety-alignment-linear-safety-separability-emerges-with-scale-and-subspace-compression-defends-against-it.

  • Graph-theoretic properties are linearly decodable from GNN embeddings at both node and graph level, with a scale-dependent collapse of global/size-related properties in task-specialized final layers. Pelletreau-Duris, van Bakel & Cochez (2024) fit linear-regression diagnostic classifiers (R2R^2) mapping GCN/GAT/GIN layer activations to dozens of local (degree, centrality measures, clustering coefficient) and global (density, diameter, #cliques, spectral radius, algebraic connectivity, small-world index) graph properties, on synthetic Grid-House graphs and the ClinTox molecular dataset. On ClinTox, size-correlated properties (#nodes/#edges) are near-perfectly linear early in a GIN (R2=1.00R^2=1.00 at layers x1-x3) but collapse to R2≈0.04R^2\approx0.04-0.070.07 by the final pre-classification layer (x8), alongside a milder but consistent decay of node-level degree R2R^2 (0.99 to 0.85) while node-level betweenness centrality stays flat (0.49-0.59) across all layers — mirroring the general pattern of task-irrelevant linear structure being compressed away in later, more task-specialized layers. A correlation of 0.992 between task accuracy and maximum probing score across architectures links probe-recoverable graph-property information directly to downstream performance. See pelletreau-duris-etal-2024-graph-theoretic-properties-are-linearly-decodable-from-gnn-embeddings-with-global-size-properties-collapsing-in-task-specialized-final-layers.

  • A single dominant PCA axis orders protein sequences from well-modeled to poorly-modeled, and viral proteins remain linearly separable from cellular proteins beyond what that axis alone predicts. Bigot, Bhasin, Park, Shakhnovich & Wang (2026) run PCA on mean-pooled final-layer embeddings from five architecturally distinct pretrained protein language models (ESM2 8M-15B, ESMC 300M-6B, ESM3 1.4B-98B, autoregressive ProGen2-base, diffusion-based EvoDiff), finding the top component — a “nativeness axis” — explains 54.3-73.1% of variance and correlates with masked-reconstruction perplexity at ρ=0.83\rho=0.83-0.960.96 (replicated within each of 10 biological groups separately, Table 3). Independently, linear probes trained on the same mean-pooled embeddings separate viral from cellular proteins at AUC 0.97-1.00 even under a homology-controlled train/test split (leave-one-family-out mean AUC >0.96>0.96), with low-false-positive-rate sensitivity (88.3-96.7% TPR at 1% FPR) far exceeding a perplexity-only classifier (26.9-39.2%) — showing the linearly separable viral signal exceeds what the dominant nativeness axis (itself tied to perplexity) alone would predict. See bigot-etal-2026-viral-proteins-reveal-a-dominant-nativeness-axis-in-protein-language-model-embedding-space.

  • A linear probe fit on SAE-decomposed, PCA-reduced residual-stream activations detects reward-hacking behavior token-by-token during generation, agreeing with an LLM judge at F1 up to 1.00 on held-out control data. Wilhelm, Wittkopp & Kao (2026) train a per-layer sparse autoencoder on the last four layers of Qwen2.5-Instruct-7B, LLaMA-3.1-8B, and Falcon3-7B (each LoRA-finetuned via GRPO on a mix of control and reward-hacking data), then fit a logistic-regression classifier on PCA-reduced, standardized SAE features to produce a per-token hack probability, averaged into a prompt-level score. Agreement with a GPT-4o judge reaches F1=1.000 on all three models’ control-only data, and F1=0.903 (Falcon), 0.961 (Llama), 0.784 (Qwen) on hack-adapter data, with mixed-ratio adapters (5-90% hack data) ranging F1=0.760-0.946 — a quantified linear-separability claim (no causal steering/ablation is performed). See wilhelm-wittkopp-kao-2026-a-linear-probe-on-sae-decomposed-pca-reduced-activations-detects-reward-hacking-token-by-token-with-up-to-perfect-agreement-with-an-llm-judge.

  • The first 1-3 components of a fixed, shared PCA basis cleanly separate figure from ground pixels across multiple architecturally distinct real pretrained vision models, validated against ground-truth segmentation masks rather than a behavioral proxy alone. Li, Wen, Song, Liu, Jing & Lee (2025) project internal activations of ViT-MAE, CLIP ViT-H/14, DINOv2, and ConvNeXt-MAE onto a shared low-dimensional PCA basis and show this axis separates figure from ground on 1,495 PASCAL VOC images checked against ground-truth masks; a complementary Top-K activation-sparsity intervention raises a texture-synthesis oddity-detection benchmark from 69.8 to 94.6 (MAE ViT) and 62.1 to 88.1 (ConvNeXt-V1). See li-etal-2025-the-first-1-3-shared-pca-components-of-self-supervised-vision-model-activations-cleanly-separate-figure-from-ground-pixels-across-vit-clip-dinov2-and-convnext-checkpoints.

  • The full Gram matrix of pairwise patch-token cosine similarities in self-supervised ViTs clusters patches by object identity, with self-supervised training producing substantially more human-aligned grouping than supervised training at matched architecture. Adeli, Ahn, Luo, Zhang, Kriegeskorte & Zelinsky (2026) compute patch-affinity/Gram-matrix structure in DINOv3, DINOv2, DINO, MAE, supervised ViT-B, and ConvNeXt-B, quantifying object-centricity via AUC against ground-truth boundaries and a grouping-accuracy benchmark: DINOv3 91.9%, DINOv2 89.0%, DINO 76.5%, MAE 80.7%, versus supervised ViT-B 70.6-72.2% and supervised ConvNeXt-B 60.0-67.4% (DINOv3-distilled ConvNeXt-B 86.7%). See adeli-etal-2026-patch-token-cosine-affinity-and-gram-matrix-structure-in-self-supervised-vits-clusters-patches-by-object-identity-with-dinov3-reaching-91-9-percent-human-aligned-grouping-accuracy.

  • MLP probes on a real pretrained generative music model’s internal activations decode seven synthetic music-theory concepts at up to 0.984 mean accuracy, varying systematically by which processing stage (audio codec vs. language-model decoder) is probed. Wei, Freeman, Donahue & Sun (2024, SynTheory) probe Jukebox and MusicGen small/medium/large, finding mean decodability of 0.984 (Jukebox LM), 0.914-0.950 (MusicGen decoder stage across sizes), but only 0.701 at the MusicGen audio-codec stage alone. See wei-freeman-etal-2024-mlp-probes-on-jukebox-and-musicgen-internal-activations-decode-seven-music-theory-concepts-at-up-to-0-984-mean-accuracy.

  • Inflectional morphology stays linearly separable across every layer of 25 real pretrained models, while lexical identity (lemma) starts linearly separable but becomes increasingly non-linearly encoded toward final layers — a depth-dependent dissociation confirmed by control-task selectivity. Li & Subramani (2025) probe BERT-Base/Large, DeBERTa-v3-Large, GPT-2 Small/Large/XL, Pythia-6.9B, OLMo-2-7B, Gemma-2-2B, Qwen2.5-1.5B, and Llama-3.1-8B (base and instruct variants), finding inflection stays linearly separable at 0.9-1.0 accuracy uniformly across depth (MLP probes add no benefit) while lemma decodability’s linear accuracy declines toward final layers even as MLP-probe accuracy declines much less; control-task selectivity confirms inflection reflects generalizable abstraction (0.4-0.6) versus lemma’s near-zero-selectivity memorization. See li-subramani-2025-inflectional-morphology-stays-linearly-separable-across-all-layers-of-25-models-while-lexical-identity-becomes-increasingly-nonlinearly-encoded-toward-the-final-layers.

  • Linear probes on hidden states substantially outperform output-level uncertainty signals at detecting when an LLM’s internal state distinguishes knowledge-recall failure from genuine truthfulness failure, and the same dissociation replicates across two model families. Cheang, Chan, Zhang & Deng (2026) probe LLaMA-3-8B and Mistral-7B-v0.3 subject-token/attention/last-layer hidden states, finding last-token-probe AUROC 0.69±0.03 (unrecognized-vs-familiar hallucination) rising to 0.93±0.01 (unfamiliar-vs-familiar), while output confidence and self-reported certainty stay far lower — accompanied by cosine-similarity and cluster-separability (Silhouette/Davies-Bouldin) evidence that “knowing” and “truthfulness” occupy measurably different regions of hidden-state space. See cheang-etal-2026-linear-probes-on-llm-hidden-states-separate-knowledge-recall-familiarity-from-truthfulness-far-better-than-output-level-confidence-signals.

  • A linear probe on hidden states detects question-unanswerability far better than a model’s own output confidence, and this “blind spot” — the model represents unanswerability internally without reporting it — does not shrink with scale, across five instruction-tuned models from three families. Wagner (2026) fits logistic-regression probes on final-prompt-token hidden states of Gemma-2-2B-it, Qwen2.5-3B/7B/14B-Instruct and Llama-3.1-8B-Instruct, finding hidden-state answerability-readout AUROC 0.97-0.99 versus only 0.54-0.67 for output-confidence-based readouts, with the gap worst on naturally occurring false-premise questions (CREPE: confidence-based signals near chance, hidden-state probe 0.69-0.77 AUROC); routing a premise-check instruction through the probe roughly triples challenge precision over the same instruction alone. See wagner-2026-a-linear-probe-on-llm-hidden-states-detects-question-unanswerability-far-better-than-output-confidence-and-this-blind-spot-does-not-shrink-with-scale.

  • Layer-wise linear probes on real pretrained VLM hidden states decode scientific-chart signal far above the model’s own end-to-end decision accuracy, quantifying an “encoded but not routed” dissociation. Kumar, Ho, Schopf, Greiner-Petter, Boudin & Aizawa (2026) find a mean-pool probe on InternVL3-8B reaches 79.0% accuracy on basic charts versus only 56.2% for the model’s own decision, and probe AUROC exceeds last-token AUROC by up to 17 points on Qwen2.5-VL-32B-Instruct (both McNemar-significant, p<0.01); the analysis is explicitly diagnostic, not causal. See kumar-etal-2026-linear-probes-on-vlm-hidden-states-decode-chart-signal-at-79-percent-accuracy-while-the-model-itself-only-reaches-56-2-percent-a-quantified-encoded-but-not-routed-dissociation.

  • Linear probes on per-head attention-contribution features decode relation type more accurately than probes on the full pooled attention state, across four real instruction-tuned LLMs. Popović & Färber (2026) decompose each attention head’s residual-stream contribution via a Direct-Logit-Attribution-style attribution and fit probes on FewRel relation classification, reaching 90.26-91.09% accuracy versus 83.65-86.79% for a full-attention-state baseline across Llama-3.2-1B/3B-Instruct, Llama-3.1-8B-Instruct and Qwen3-4B-Instruct. See popovic-farber-2026-per-head-attention-contribution-linear-probes-decode-relation-type-at-90-91-percent-accuracy-across-four-instruction-tuned-llms-exceeding-full-attention-state-probes.

  • Layer-wise linear probing of periodic-table/chemistry facts shows middle layers encode element attributes continuously and overlapping, while deeper layers sharpen the same attributes into more categorically separated clusters — alongside a 3D-PCA spiral trajectory tracking atomic-number ordering. Lei & Cooper (2025) probe Llama-3-8B and Llama-3.1-70B-Instruct, reporting per-property spiral-fit consistency scores up to 0.90 for the PCA trajectory and confirming the continuous-to-categorical depth transition via cosine-similarity analysis of learned attribute directions. See lei-cooper-2025-a-3d-pca-spiral-in-llama-hidden-states-aligns-with-the-periodic-tables-conceptual-structure-and-linear-probes-show-middle-layers-superimpose-related-attributes-while-deeper-layers-separate-them.

  • Linear and MLP probes decode a task-relevant spatial “cognitive map” from a real LLM agent’s pre-reasoning activations, validated behaviorally via policy-agreement rather than probe accuracy alone. Arghal, Chen, Dalton, Kortukov, McNamara, Nalmpantis, Nirvaan, Sarti & Giulianelli (2026, ICML) probe GPT-OSS-20B’s layer-15 activations in a 2D grid-world navigation task, finding 82.5% average policy-agreement between the agent’s action and the decoded cognitive map, with post-reasoning activations reorganized away from this spatial code. See arghal-etal-2026-linear-and-mlp-probes-decode-a-grid-position-cognitive-map-from-gpt-oss-20bs-pre-reasoning-activations-and-policy-agreement-with-the-decoded-map-averages-82-5-percent.

  • Linear probing classifiers on real pretrained protein language model embeddings recover contact maps, binding sites and secondary structure, converging with an independent attention-alignment measurement on the same models. Vig, Madani, Varshney, Xiong, Socher & Rajani (2020) analyze TapeBert, TAPE-ALBERT and XLNet, finding individual attention heads concentrate on binding sites far above background frequency (e.g. one head at 49%, Bonferroni-corrected p<0.00001) while a separate embedding-level linear probe quantifies the same structures via precision@L/5, precision@L/20 and F1. See vig-etal-2020-linear-probes-on-real-pretrained-protein-lm-embeddings-recover-contact-maps-binding-sites-and-secondary-structure-and-individual-attention-heads-align-with-the-same-structures-far-above-background-frequency.

  • A middle-layer attention-head group linearly decodes another agent’s belief status above 80% validation accuracy, dissociable from the model’s own belief representation, and manipulating it causally changes Theory-of-Mind task performance. Zhu, Zhang & Wang (2024, ICML) probe Mistral-7B on third-person false-belief narratives, generalizing the causal steering result across social-reasoning tasks with different causal-inference patterns. See zhu-etal-2024-a-middle-layer-attention-head-group-in-mistral-7b-linearly-decodes-another-agents-belief-status-above-80-percent-validation-accuracy-and-manipulating-it-causally-changes-theory-of-mind-performance.

  • A causally-present backdoor trigger signal can occupy a subspace orthogonal to a linearly-probed language-identity direction, so a linear probe fails to detect it even while activation patching confirms it is driving the model’s behavior. Kulumba, Antoun, Lasnier, Sagot & Seddah (2026) show this in Gaperon-8B’s language-switching backdoor: the probe reads “English” throughout mid-to-late layers while the trigger is causally present, only converted to French output by a final-layer MLP accounting for ~63% of the total causal effect — sharpening the general distinction between “linearly probeable” and “causally present” sharper than in most probing papers in this map. See kulumba-etal-2026-a-backdoor-trigger-signal-in-gaperon-8b-persists-in-a-subspace-orthogonal-to-a-linearly-probed-language-identity-direction-throughout-mid-late-layers-despite-being-causally-present.

  • Raw attention matrices from a real pretrained protein language model, read out through a single linear map, recover contact maps at accuracy competitive with a far more expensive combinatorial mutation-sweep method. Thorstenson (2026) shows a learned linear read-out from ESM-2 (35M/150M/650M/3B) and ESMC-600M attention matrices to residue-residue contact maps matches the categorical Jacobian method’s accuracy, despite the categorical Jacobian requiring an expensive per-protein sweep of masked-residue mutations while this read-out needs only a single forward pass — evidence that contact information is already linearly separable within raw attention patterns, rather than requiring nonlinear, mutation-based probing. See esm-attention-maps-are-linearly-readable-as-protein-contact-maps.

  • Attribute-object binding is linearly decodable within a single modality of a real pretrained vision-language model but collapses cross-modally, and a lightweight learned linear map closes the gap. Koishigarina, Uselis & Oh (2026) probe CLIP (ViT-B/32, ViT-B/16, ViT-L/14) for attribute-object binding, finding near-ceiling linear-probe accuracy uni-modally (up to 1.0 train / 0.91 test) but near-chance cross-modal accuracy (0.51-0.52) at baseline, restored to 0.91-1.00 by a 262K-parameter learned linear transform — isolating the deficit to cross-modal alignment rather than missing linear structure in either modality. See clip-attribute-object-binding-is-linearly-decodable-uni-modally-but-collapses-cross-modally-until-a-linear-map-restores-it.

  • A linear probe on reasoning-model hidden states predicts eventual answer correctness, including for answers not yet stated, well above chance across a family of distilled reasoning models. Zhang, Chen, Pan, Zhao, Panda, Li & He (2025) find hidden states at intermediate-answer positions in six reasoning models (DeepSeek-R1-Distill-Llama-8B/70B, DeepSeek-R1-Distill-Qwen-1.5B/7B/32B, QwQ-32B) predict correctness at ROC-AUC above 0.7, with a grid search showing many probes converge to a purely linear one; the effect is much weaker in a non-reasoning baseline (Llama-3.1-8B-Instruct), and the probe deployed as an early-exit verifier cuts inference tokens 24% with no accuracy loss. See reasoning-model-hidden-states-linearly-encode-answer-correctness-before-the-answer-is-stated.

  • Lightweight vision-language models linearly encode object/agent presence from mid-vision-encoder through the final LLM layers, but never linearly encode orientation, and steering the probe direction causally confirms it drives behavior. Theodoridis et al. (2026) probe Ovis2.5-2B, InternVL3.5-2B, and VST-3B on CARLA-generated counterfactual driving scenes, finding presence is explicitly linearly separable at every stage while orientation never is, and validate the presence direction’s causal role via activation steering, distinguishing “perceptual failure” from “cognitive failure” cases. See lightweight-vlms-linearly-encode-object-presence-but-never-orientation-and-steering-the-probe-direction-causally-confirms-it.

  • A linear probe on Qwen3-32B’s residual stream decodes the tool-call dependency graph in an LLM agent’s trajectory at AUROC 0.869, and activation patching shows the representation causally propagates rather than being read out passively. Sun & Kazakov (2026) train an edge probe recovering which earlier tool-call output feeds which later call’s arguments, decisively above random-label, positional-only, and n-gram surface-form controls, with per-layer activation patching shifting later-layer predictions toward a donor trajectory’s structure. See a-linear-probe-decodes-tool-call-dependency-structure-from-qwen3-32bs-residual-stream-with-auroc-0-869-and-activation-patching-shows-the-representation-causally-propagates.

  • A linear probe on pre-generation hidden states predicts whether an LLM agent needs to call a tool at AUROC 0.89-0.96 across six models of varying family and size, including on models whose own generation catastrophically fails to express this knowledge. Sun, Liu, Yan, Wang & Weng (2026) show Reason-then-Act prompting can collapse tool-calling accuracy (Llama-3.1-8B-Instruct 79.5% to 31.2%) while a simple linear probe on the same models’ hidden states still linearly decodes tool necessity at AUROC >0.9, dissociating internal knowledge from expressed behavior. See linear-probes-decode-whether-a-tool-call-is-needed-at-auroc-0-89-to-0-96-across-six-llms-of-varying-family-and-size-including-models-whose-own-generation-fails-to-express-this-knowledge.

  • Language-conditioned linear probes and neighborhood-based mutual information reveal Indic-district geography as linearly decodable structure nested hierarchically inside each language’s embedding subspace. Kumar J et al. (2026) find fine-tuned Whisper-base and Wav2Vec2.0-base achieve mean F1 91.29/87.14 under joint language-district supervision, with a Normalized Conditional Mutual Information analysis showing an asymmetric hierarchical nesting of district clusters inside language-level structure. See fine-tuned-whisper-and-wav2vec2-encode-indic-dialect-geography-as-a-linearly-decodable-hierarchical-geometry-nested-inside-each-languages-embedding-subspace.

  • TCAV-style linear probes detect chess concepts emerging at different points during training in two custom self-play RL chess agents, replicating AlphaZero’s concept-probing methodology at toy scale. Hammersborg & Strumke (2022) show material-advantage and mate-threat concepts are linearly decodable from initialization in a ResNet-based agent (due to skip connections) while plateauing after ~100 iterations in a smaller CNN agent. See tcav-style-linear-probes-detect-chess-concepts-like-material-advantage-and-mate-threat-emerging-at-different-training-points-in-two-custom-self-play-rl-chess-agents.

  • Linear probes on residual-stream activations detect motivated reasoning both before and after chain-of-thought generation, with a U-shaped hint-recovery signal showing the model re-engages a hint internally even when CoT text never mentions it. Mirtaheri & Belkin (2026) find pre-generation probes match a full-CoT-trace external judge, and post-generation probes outperform it, across three models. See linear-probes-on-residual-stream-activations-detect-motivated-reasoning-both-before-and-after-cot-generation-with-a-u-shaped-hint-recovery-signal-across-cot-tokens.

  • A lightweight covariance-pooling probe on forecasting-LLM activations halves calibration error versus verbalized confidence and acts as a “lie detector,” correctly predicting belief-change direction in 83.6% of cases including 107 where CoT text shows no change at all. Sarfati et al. (2026) show this linear/bilinear readout holds up out-of-distribution where verbalized confidence collapses (AUROC 0.587). See a-linear-covariance-pooling-probe-on-forecasting-llm-activations-halves-calibration-error-versus-verbalized-confidence-and-detects-107-stealth-cases-where-cot-shows-no-change.

  • Logistic-regression probes on the residual stream detect parametric-vs-contextual knowledge conflict and predict, before generation, which source the model will rely on. Zhao et al. (2024) find detection AUROC/AUPRC peaks around layer 13-14 of Llama3-8B and Llama2-7B, with source-prediction accuracy improving up to layer 16-17, after the conflict-detection signal peaks. See logistic-regression-probes-on-the-residual-stream-detect-parametric-vs-contextual-knowledge-conflict-and-predict-which-source-the-model-will-rely-on-before-generation-with-a-distinct-skewness-signature.

  • Pretraining-time outlier dimensions persist through GLUE fine-tuning, and a single principal outlier dimension alone completes downstream tasks with minimal accuracy loss, sometimes beating the full fine-tuned classifier head. Rudman, Chen & Eickhoff (2023) show this across eight encoder/decoder models, with accuracy drops as low as 0-3.4% for several models when using only the 1-D outlier subspace. See pretraining-outlier-dimensions-persist-through-fine-tuning-and-a-single-principal-outlier-dimension-alone-can-complete-glue-tasks-with-minimal-accuracy-loss-across-eight-encoder-and-decoder-models.

  • A layer-localized linear probe decodes which moral framework a model is reasoning under mid-trajectory, beating a training-prior baseline by 13.8-22.6% lower KL divergence. Huang, Kwak & An (2026) find this in Llama-3.3-70B and Qwen2.5-72B, cross-referenced from linear-direction for its causal steering component. See a-layer-localized-linear-probe-direction-for-moral-framework-decodes-which-ethical-framework-a-model-is-using-mid-reasoning-and-steering-along-it-shifts-framework-integration-drift.

  • Linear probes decode future planning steps in a Blocksworld task from a single forward pass, with decoding accuracy decaying smoothly with prediction distance and tracking a nonlinear-probe control. Men et al. (2024) find a linear probe predicts the 6th-step-ahead decision from the 1st-step representation at 0.51 accuracy in fine-tuned Llama-2-7b-chat and Vicuna-7B, with linear and nonlinear probes tracking the same decay trend. See linear-probes-on-fine-tuned-llm-residual-streams-decode-future-planning-steps-in-a-blocksworld-task-with-accuracy-decaying-smoothly-with-prediction-distance-tracking-nonlinear-probes.

  • A linear probe direction separates real-world evaluation from deployment prompts, and the same probe classifies artificial safety evaluations as inauthentic. Nguyen, Hoang, Attubato & Hofstätter (2025) demonstrate this in Llama-3.3-70B-Instruct, cross-referenced from linear-direction for its causal steering component. See a-linear-probe-direction-separates-real-world-evaluation-from-deployment-prompts-in-llama-3-3-70b-and-the-model-classifies-artificial-safety-evaluations-as-inauthentic-evidencing-an-internal-eval-deploy-axis.

  • Layerwise linear probes on real pretrained video foundation models find intuitive-physics information most decodable in a predictive joint-embedding architecture, competitively decodable in a masked-reconstruction architecture, and weaker but non-trivial in a video-diffusion generator, peaking at intermediate-to-late depth. Punzo, Caselli, Pantelidis, Massafra, Lo Sardo & Salehi (2026) probe V-JEPA, VideoMAE, and LTX-Video on the IntPhys2 and Minimal Video Pairs benchmarks, confirming the probed signal reflects genuine temporal structure via a frame-shuffling control that substantially degrades performance. See layerwise-linear-probing-shows-video-foundation-models-vary-in-how-linearly-decodable-intuitive-physics-is-with-video-jepa-strongest.

  • Approximately inverting a video diffusion transformer’s deterministic sampling process to recover intermediate states makes physical plausibility linearly decodable well above representation-learning baselines probed on their own forward pass. Esmati, Nath, Hofmann, Nowrouzezahrai, Ebrahimi Kahou & Mirmehdi (2026) invert three checkpoints (WAN-1.3B, CogVideoX-2B, LTX-2B) via backward velocity-field integration, reaching ~81.27% average linear-probe accuracy on IntPhys/InfLevel and exceeding V-JEPA2 and VideoMAE baselines — evidence that a diffusion model encodes more physical structure internally than its generated output or ordinary forward-pass activations reveal. New method diffusion-inversion-probing. See diffusion-inverted-video-transformer-states-are-linearly-decodable-for-physical-plausibility-and-outperform-representation-learning-baselines.

  • Linear probes across 78 molecular substructures, run on both pretrained and randomly-initialized chemical language models, isolate which substructure-encoding is a genuine product of pretraining versus an architecture-driven default. Karnysheva, Klakow & Lee (2026) find pretraining generally improves substructure awareness in upper layers across eight pretrained CLMs, while six randomly-initialized counterparts already encode ring structures well from layer 1. See linear-probes-across-78-molecular-substructures-show-pretraining-improves-chemical-language-model-substructure-awareness-especially-in-upper-layers.

  • A Fisher-discriminant-style inter-to-intra scatter ratio, computed per attention head in a frozen real MLLM, quantifies normal/anomalous representation-manifold entanglement and selects heads for a targeted causal intervention. Cai, Li, Duan, He & Zhai (2026, SteerVAD) apply this Representational Separability Analysis (RSA) score to InternVL3-8B, using it to select the most-entangled heads for anisotropic-scaling steering, reaching 87.15% AUC on UCF-Crime video-anomaly-detection with ~1% of the training data of a fully fine-tuned baseline. New method rsa-inter-intra-scatter-ratio. See cai-etal-2026-per-head-fisher-ratio-manifold-entanglement-scores-select-targeted-anisotropic-steering-that-rectifies-frozen-mllm-anomaly-detection-manifolds.

  • Foundational: the paper that introduced linear classifier probing, showing linear separability of real trained CNN features increases monotonically with depth. Alain & Bengio (2016) fit an independent linear probe at each layer of trained Inception-v3 and ResNet-50, finding probe accuracy rises monotonically with depth — the origin of the linear-probing methodology used throughout this map, with the authors themselves flagging probe accuracy as only a lower bound on the information actually present. See alain-bengio-2016-linear-classifier-probes-fit-independently-at-every-layer-of-inceptionv3-and-resnet-50-show-linear-separability-of-learned-features-increases-monotonically-with-depth.

  • MLP probes on real LLM hidden states form tight, well-separated emotion clusters that tighten with scale and peak in probe accuracy at 50-75% relative depth, not the final layer. Zhang & Zhong (2025) probe Qwen3 (0.6B-8B) and LLaMA-3 (1B-8B), finding 7-class emotion clusters visualized via PCA+KDE become “tight, well-separated” only for the larger models, with probe accuracy rising from chance at layer 0 to a peak of 0.78-0.80 at 50-75% depth — “the network’s middle layers contain the strongest and most distinct representations of emotion” — and signal persisting, with emotion-specific decay, across up to 400 generated tokens. See zhang-zhong-2025-mlp-probes-on-real-qwen3-and-llama-3-hidden-states-form-tight-well-separated-emotion-clusters-that-peak-at-50-75-percent-depth-not-the-final-layer.

  • A real trained next-state-prediction world model linearly separates “which object changed” from an activation-difference vector, despite having no architecturally imposed per-object slots. Saanum, Schulze Buschoff, Dayan & Schulz (2024) train real CNN-encoder Contrastive World Models on rendered multi-object video (Cubes, 3-body physics, Multi-dSprites, MOVi-simple, MOVi-A) and fit an L1-regularized linear classifier on the difference vector between two consecutive frames’ encodings to predict which single object changed, reaching near-100% held-out accuracy on Cubes/3-body physics and about 70% (well above chance) on the harder Multi-dSprites/MOVi settings, while untrained encoders and a static (non-predictive) contrastive baseline stay near chance — isolating next-state prediction itself, not architecture alone, as what drives the emergent linear decodability. The same representation is not factorized into clean per-object slots the way an explicit slot-based baseline is (shared unit populations encode transformation type across different objects), making this a case of linearly separable-yet-entangled structure rather than either purely distributed or purely modular. See saanum-etal-2024-a-real-trained-contrastive-world-model-linearly-decodes-which-object-changed-from-difference-vectors-despite-having-no-imposed-object-slots.

  • Layerwise linear probes mapping where honesty/deception is decodable in real LLMs drift sharply during real RLVR training, exposing representation drift rather than deliberate evasion. Taufeeque, Heimersheim, Gleave & Cundy (2026, The Obfuscation Atlas) train per-layer logistic-regression probes on residual-stream activations to build a layerwise “atlas” of deception decodability, then run real LoRA+GRPO RLVR training (MBPP-Honeypot coding environment, where reward hacking naturally emerges) on real Llama-3-8B-Instruct, Gemma-3-12B/27B-IT, and Llama-3-70B-Instruct. Pre-RL probe AUC is near-ceiling (0.991-1.000); after RLVR training without any detector penalty, AUC on off-domain deception probes collapses (e.g. 0.991->0.376, 0.996->0.655) while on-domain MBPP-probe AUC stays at 1.000 — the decodability the probe relied on for out-of-domain generalization drifts away under real training, not because the model learns to evade detection (with a detector penalty added, genuinely evasive deceptive behavior instead reaches 96.4% of seeds). See taufeeque-etal-2026-layerwise-linear-probes-for-honesty-decodability-in-real-llms-drift-sharply-off-domain-under-real-rlvr-training-while-on-domain-decodability-stays-near-ceiling.

  • A single frozen linear classifier trained on one astronomical survey’s embeddings transfers zero-shot to a different telescope’s images with no accuracy loss, evidencing survey-invariant structure in a real omnimodal foundation model. Parker, Lanusse, Shen et al. (2025, AION-1) train a 10-way linear/softmax morphology classifier on frozen AION-1 embeddings of Legacy Survey galaxy images (Galaxy Zoo-10 classes) and apply the identical, un-fine-tuned head to Hyper Suprime-Cam images from a different telescope, filter set and pixel scale (overlapping galaxies excluded to prevent leakage): zero-shot HSC accuracy (84.15-85.91%) matches or exceeds native-domain training accuracy (83.95-87.16%) across three model sizes (Table 6). No causal intervention is performed, and the paper is otherwise benchmark-dominated across many tasks; this cross-survey linear transfer is the one genuine emergent-structure finding. See parker-etal-2025-a-frozen-linear-morphology-classifier-fit-on-aion-1-embeddings-transfers-zero-shot-across-telescope-surveys.

  • Environmental variables never used during pretraining are linearly decodable from frozen geospatial self-supervised encoders, with the strength of decodability depending on which self-supervised objective was used, not just on downstream task accuracy. Mocharla & Patel (2026) fit ridge-regression (linear) and MLP (nonlinear) probes for five ERA5 environmental variables (temperature, precipitation, solar radiation, pressure, soil water) on real DINO/MAE/MoCo ViT-S/16 encoders trained identically on SSL4EO Sentinel-1/2 imagery, plus several public geospatial foundation models. DINO shows the strongest ERA5 decodability (linear R2=0.50R^2=0.50, MLP R2=0.68R^2=0.68) versus MAE/MoCo near-random despite similar downstream segmentation mIoU; linear-probe R2R^2 correlates more strongly with downstream agricultural-task utility than MLP R2R^2 (ρ=0.75\rho=0.75 vs. 0.630.63). Intrinsic geometry metrics (effective rank, uniformity) correlate with probe performance (ρ≈0.40\rho\approx0.40-0.450.45); purely associative, the authors explicitly state the analysis is not causal. See mocharla-patel-2026-environmental-variables-never-seen-in-pretraining-are-linearly-decodable-from-frozen-geospatial-ssl-encoders-with-architecture-dependent-strength.

  • A real self-supervised speech model trained on longitudinal dolphin recordings organizes its learned quantization codebook by whistle category and individual identity, with a temporal-shuffling ablation confirming dependence on genuine temporal structure. Semenzin, Mustun, Dessì, Orhan, Emanuelli, Lakretz, de Polavieja & Sumbre (2026) train Dolph2Vec, a Wav2Vec2.0-architecture model adapted for 44.1kHz audio, on ~180,000 whistles (100 hours) from 5 known bottlenose dolphins recorded over 5 real years. The codebook’s specialization for signature-whistle categories is quantified via conditional entropy and mutual information (training reduces entropy 2.13->1.85 and raises MI 0.43->0.70 vs. a random-init baseline); UMAP-projected, GMM-clustered embeddings score ARI=0.3565 and NMI=0.4226 against ground-truth whistle labels, beating AVES-bio/BioLingual baselines. A temporal-shuffling ablation on the feature-encoder output drops downstream classification accuracy from 82.0% to 75.1%. See semenzin-etal-2026-a-real-wav2vec2-style-model-trained-on-longitudinal-dolphin-recordings-organizes-its-codebook-by-whistle-category-and-individual-identity.

  • Frozen real genomic language models show task-dependent linear accessibility of biological signal, quantified by silhouette, kNN and Fisher-separation geometry, not just probe accuracy. Datta, Shatabda & Rahman (2026) probe five real frozen genomic language models (DNABERT-2, Nucleotide Transformer, HyenaDNA, GENERATOR-v2, Omni-DNA) with logistic-regression (linear), RBF-SVM and MLP probes plus quantitative embedding-geometry metrics. Promoter-recognition signal is highly linearly accessible (frozen-probe recovery of fine-tuned performance: 98/95/100/98% for HyenaDNA/NT/GENERATOR-v2/Omni-DNA), while splice-site signal is markedly less accessible (55-88% recovery), with Nucleotide Transformer’s own embedding geometry showing the same split quantitatively (Promoter-All silhouette=0.044/kNN=0.883/linear-separability=0.917 vs. Splice-All silhouette=-0.011/kNN=0.465/linear-separability=0.597). Layer-wise probing shows intermediate layers (layer 20: ACC=0.639) outperform the final layer (layer 24: ACC~0.60); causal validation is via input-level in-silico mutagenesis, not internal activation patching. See datta-etal-2026-frozen-real-genomic-language-models-show-task-dependent-linear-accessibility-of-biological-signal-quantified-by-silhouette-knn-and-fisher-separation-geometry.

  • Sparse-code decomposition of real neural audio codecs splits accent-predictive information between activation magnitude and position, with the split reversing between acoustic- and phonetic-oriented codecs. Wang, Feng, Kommineni, Lertpetchpun, Yi, Shi & Narayanan (2026) train TopK SAEs on real EnCodec, DAC, SpeechTokenizer and Mimi, then fit logistic-regression probes on the full sparse code, its position-only pattern, and its magnitude-only pattern for accent classification. For DAC (acoustic-oriented), magnitude-only features preserve more accent information than position-only (ΔF1 ~-5.0 to -5.1% vs. ~-9.8 to -12.0% at 5% sparsity); for SpeechTokenizer (phonetic-oriented, distilled from HuBERT/WavLM), the pattern reverses (position-only ~-4.5 to -6.3% vs. magnitude-only ~-7.1 to -7.2%). Purely correlational, no steering intervention performed. See wang-etal-2026-sparse-autoencoder-decompositions-of-real-neural-audio-codecs-encode-accent-in-activation-magnitude-or-position-depending-on-codec-type.

Exercises

Base

  1. Classes A={(1,1),(2,2)}A=\{(1,1),(2,2)\}, B={(−1,−1),(−2,−1)}B=\{(-1,-1),(-2,-1)\}. Find a w,bw,b that linearly separates them.
Solution

Take w=(1,1)w=(1,1), b=0b=0. Check: w⋅(1,1)+0=2>0w\cdot(1,1)+0=2>0, w⋅(2,2)+0=4>0w\cdot(2,2)+0=4>0 (both in AA, both positive); w⋅(−1,−1)+0=−2<0w\cdot(-1,-1)+0=-2<0, w⋅(−2,−1)+0=−3<0w\cdot(-2,-1)+0=-3<0 (both in BB, both negative). This (w,b)(w,b) separates AA from BB.

  1. Two classes each consist of a single point: A={a}A=\{a\}, B={b}B=\{b\} with a≠ba\neq b. Are they always linearly separable?
Solution

Yes — any two distinct points are always separable by the hyperplane perpendicular to a−ba-b passing through the midpoint (a+b)/2(a+b)/2: take w=a−bw=a-b, and c=w⋅a+b2c = w\cdot\frac{a+b}{2}, so the hyperplane is {x:w⋅x=c}\{x:w\cdot x=c\}, with aa and bb strictly on opposite sides since w⋅a−c=w⋅(a−b)/2=∥a−b∥2/2>0w\cdot a - c = w\cdot(a-b)/2 = \|a-b\|^2/2>0 and symmetrically w⋅b−c<0w\cdot b - c<0.

Middle

  1. Prove that if A,BA,B are linearly separable with margin m>0m>0, then every point of AA is at Euclidean distance at least mm from every point of BB… or show this is false with a counterexample.
Solution

False in general — the margin is the distance from each point to the hyperplane, not to points of the other class directly. Counterexample: let the hyperplane be the yy-axis in R2\mathbb{R}^2 (w=(1,0),b=0w=(1,0),b=0), with A={(0.1,100)}A=\{(0.1, 100)\} and B={(−0.1,100)}B=\{(-0.1,100)\}. Both points are at distance 0.10.1 from the hyperplane (so margin =0.1=0.1 if these are the closest points), but the distance between the two points themselves is only 0.20.2 — actually in this example distance between points (0.20.2) does happen to be close to 2×2\timesmargin; to see a genuine failure, consider points nearly on the hyperplane but far apart along it: A={(0.1,1000)}A=\{(0.1, 1000)\}, B={(−0.1,−1000)}B=\{(-0.1,-1000)\}, margin still 0.10.1, but the two points are enormously far apart (distance ≈2000\approx 2000) — so margin lower-bounds nothing about upper distance, and more importantly margin does not claim to be a lower bound on cross-class point distances in general either, since points from AA and BB near the same location along the hyperplane but on opposite sides can be as close as (in the limit) twice the margin — margin is fundamentally about proximity to the boundary, not a pairwise-distance guarantee between the classes.

  1. Prove the “only if” direction of the convex-hull characterization: if A,BA,B are linearly separable, their convex hulls are disjoint.
Solution

Suppose w,bw,b separate A,BA,B as in the definition. For any point pp in the convex hull of AA, p=∑iciaip=\sum_i c_i a_i with ai∈Aa_i\in A, ci≥0c_i\geq0, ∑ici=1\sum_i c_i=1. Then w⋅p+b=∑ici(w⋅ai)+b=∑ici(w⋅ai+b)−∑icib+bw\cdot p+b = \sum_i c_i(w\cdot a_i) + b = \sum_i c_i(w\cdot a_i+b) - \sum_ic_i b+b… more directly: w⋅p+b=∑ici(w⋅ai+b)w\cdot p + b = \sum_i c_i(w\cdot a_i+b) using ∑ici=1\sum_i c_i=1 to distribute bb. Since each w⋅ai+b>0w\cdot a_i+b>0 and ci≥0c_i\geq0 with at least one ci>0c_i>0, the weighted sum is >0>0. So every point of conv(A)\text{conv}(A) satisfies w⋅p+b>0w\cdot p+b>0; symmetrically every point of conv(B)\text{conv}(B) satisfies w⋅p+b<0w\cdot p+b<0. Since no point can satisfy both, conv(A)∩conv(B)=∅\text{conv}(A)\cap\text{conv}(B)=\emptyset.

Pro

  1. Prove that linear separability is preserved under any invertible affine transformation T(x)=Mx+tT(x) = Mx+t (MM invertible) applied to both classes — i.e. if A,BA,B are separable, so are T(A),T(B)T(A), T(B).
Solution

Given w,bw,b separating A,BA,B: for x∈Ax\in A, w⋅x+b>0w\cdot x+b>0. We want w′,b′w', b' such that w′⋅T(x)+b′>0w'\cdot T(x)+b'>0 for all such xx, i.e. w′⋅(Mx+t)+b′=(M⊤w′)⋅x+(w′⋅t+b′)>0w'\cdot(Mx+t)+b' = (M^\top w')\cdot x + (w'\cdot t+b')>0. Matching to the original condition, set M⊤w′=wM^\top w' = w, i.e. w′=(M⊤)−1ww' = (M^\top)^{-1}w (using invertibility of MM, hence of M⊤M^\top), and w′⋅t+b′=bw'\cdot t+b'=b, i.e. b′=b−w′⋅tb'=b-w'\cdot t. Then w′⋅T(x)+b′=w⋅x+bw'\cdot T(x)+b' = w\cdot x+b exactly, which is >0>0 for x∈Ax\in A and <0<0 for x∈Bx\in B by assumption — so (w′,b′)(w',b') separates T(A),T(B)T(A),T(B).

  1. Let A={(−1,0)}A=\{(-1,0)\}, B={(1,0)}B=\{(1,0)\}, C={(0,0.5),(0,−0.5)}C=\{(0,0.5),(0,-0.5)\} in R2\mathbb{R}^2. Verify A,B,CA,B,C are pairwise linearly separable, then show CC is not linearly separable from A∪BA\cup B by any hyperplane through the origin — pairwise separability of three classes does not imply any one of them is separable from the union of the other two.
Solution

Pairwise: AA vs BB — take w=(−1,0),b=0w=(-1,0), b=0: gives 1>01>0 on AA, −1<0-1<0 on BB. AA vs CC — both points of CC have x=0>−1x=0>-1, so w=(1,0),b=−0.5w=(1,0), b=-0.5 gives −1−(−0.5)-1-(-0.5)… more simply w=(−1,0),b=0.5w=(-1,0), b=0.5: on AA, 1+0.5=1.5>01+0.5=1.5>0; on CC, 0+0.5=0.5>00+0.5=0.5>0 — same sign, wrong; instead take w=(1,0),b=0.5w=(1,0),b=0.5: AA: −1+0.5=−0.5<0-1+0.5=-0.5<0; CC: 0+0.5=0.5>00+0.5=0.5>0. Separated (labels swapped, which is immaterial). BB vs CC: symmetric, w=(−1,0),b=0.5w=(-1,0),b=0.5 gives BB: −1+0.5=−0.5<0-1+0.5=-0.5<0, CC: 0.5>00.5>0. Separated. So all three pairs are separable.

Now suppose, for contradiction, a hyperplane through the origin (b=0b=0 WLOG for this construction, though the argument below in fact works for general bb too) with normal w=(w1,w2)w=(w_1,w_2) separates CC from A∪BA\cup B. Say CC is on the negative side and A∪BA\cup B on the positive side (the other labeling is symmetric). Then:

  • From A=(−1,0)A=(-1,0): −w1+b>0⇒b>w1-w_1 + b > 0 \Rightarrow b > w_1.
  • From B=(1,0)B=(1,0): w1+b>0⇒b>−w1w_1+b>0 \Rightarrow b>-w_1.
  • From C=(0,0.5)C=(0,0.5): 0.5w2+b<0⇒b<−0.5w20.5w_2+b<0 \Rightarrow b<-0.5w_2.
  • From C=(0,−0.5)C=(0,-0.5): −0.5w2+b<0⇒b<0.5w2-0.5w_2+b<0 \Rightarrow b<0.5w_2.

From the first two: b>max⁡(w1,−w1)=∣w1∣≥0b > \max(w_1,-w_1) = |w_1| \geq 0. From the last two: b<min⁡(−0.5w2,0.5w2)=−0.5∣w2∣≤0b < \min(-0.5w_2, 0.5w_2) = -0.5|w_2| \leq 0. Combining: 0≤∣w1∣<b<−0.5∣w2∣≤00 \leq |w_1| < b < -0.5|w_2| \leq 0, i.e. 0<00 < 0 — a contradiction. So no such w,bw,b exists: CC cannot be separated from A∪BA\cup B, even though every pair among A,B,CA,B,C is separable. Geometrically, CC‘s two points straddle the segment ABAB (one above the xx-axis, one below), so any line separating them from {A,B}\{A,B\} would have to keep both AA and BB on one side while keeping points just above and just below the axis on the other — impossible once A,BA,B sit on the axis itself, between CC‘s two points in the relevant sense.

Found in (65 observations · 65 families)

Llama

Understanding Moral Reasoning Trajectories in LLMs: Toward Probing-Based Explainability (2026)measured

A moral-framework probe decodes and steers mid-reasoning ethics

Details

Huang, Kwak & An train linear probes on last-token residual-stream states across all 81 layers of Llama-3.3-70B and Qwen2.5-72B to predict a 5-dimensional moral-framework distribution [huang-etal-2026-understanding-moral-reasoning-trajectories-in-llms] Framework encoding localizes to model-specific layers (63/81 for Llama-3.3-70B, 17/81 for Qwen2.5-72B), achieving 13.8-22.6% lower KL divergence than a training-prior baseline [huang-etal-2026-understanding-moral-reasoning-trajectories-in-llms] Steering along the probe direction modestly reduces the Framework Drift Rate (6.7% at Llama layer 6; 8.9% at Qwen layer 1) [huang-etal-2026-understanding-moral-reasoning-trajectories-in-llms] Probing and steering are restricted to the two open-weight models, while closed models (GPT-5, GPT-4o family) are evaluated only behaviorally [huang-etal-2026-understanding-moral-reasoning-trajectories-in-llms]

models: Llama 3.3 70B Instruct · method: Linear probing, Causal interventions (steering)
Probing and Steering Evaluation Awareness of Language Models (2025)measured

An eval/deploy probe generalizes but its steering direction is inert

Details

Nguyen et al. train linear probes on Llama-3.3-70B-Instruct activations that separate real-world evaluation prompts from deployment prompts (AUROC ~0.82) [nguyen-etal-2025-probing-and-steering-evaluation-awareness-of-language-models] The same probes classify current artificial safety evaluations as test-like (Insider Trading 100%, Sandbagging 94.2%, Roleplaying 80.6%), evidencing an internal eval/deploy axis [nguyen-etal-2025-probing-and-steering-evaluation-awareness-of-language-models] Steering along the probe direction had negligible behavioral effect (<5% sandbagging recovery), so the direction is not behaviorally load-bearing [nguyen-etal-2025-probing-and-steering-evaluation-awareness-of-language-models] Only a prompt-suffix intervention produced meaningful recovery (83%), while SAE-feature steering reached at most about 25% [nguyen-etal-2025-probing-and-steering-evaluation-awareness-of-language-models]

models: Llama 3.3 70B Instruct · method: Linear probing, Causal interventions (steering)
Proxy Reward Internalization and Mechanistic Exploitation: A Learned Precursor to Reward Hacking and Its Generalization (2026)measured

Diff-in-means reward-hacking directions forecast and causally control misalignment

Details

Beigi, Jin & Huang extract three diff-in-means directions (context-sycophantic-agreement, premise-rejection, evaluator-rejection) during RL training of Qwen2.5-Coder-14B-Instruct and track their projection score across checkpoints [beigi-jin-huang-2026-proxy-reward-internalization-and-mechanistic-exploitation] Each direction is linearly decodable (AUROC 0.85-0.90) and a linear fit from probe score forecasts future out-of-domain misalignment (R^2=0.77), preceding the misalignment rise by roughly 45 training steps [beigi-jin-huang-2026-proxy-reward-internalization-and-mechanistic-exploitation] Joint ablation of the three directions lowers the hack rate by 26 points while preserving coding accuracy, and injection along them increases hacking [beigi-jin-huang-2026-proxy-reward-internalization-and-mechanistic-exploitation] The probe methodology generalizes to Qwen2.5-Coder 1.5B/7B/32B, Llama-3.1-8B-Instruct, and OLMo-7B [beigi-jin-huang-2026-proxy-reward-internalization-and-mechanistic-exploitation]

models: Llama-3.1-8B-Instruct · method: Difference-in-means direction
Do LLMs Really Know What They Don't Know? Internal States Mainly Reflect Knowledge Recall Rather Than Truthfulness (2026)measured

Probes separate knowledge familiarity from truthfulness better than confidence

Details

Cheang et al. fit logistic-regression probes on subject-token, attention, and last-layer hidden states of LLaMA-3-8B (main) and Mistral-7B-v0.3 (replication) [cheang-etal-2026-do-llms-really-know-what-they-dont-know-internal-states-mainly-reflect-knowledge-recall-rather-than-truthfulness] A last-token probe reaches AUROC 0.69 distinguishing attributable from unattributable hallucination but 0.93 distinguishing unfamiliar from familiar entities; subject and attention probes show the same gap [cheang-etal-2026-do-llms-really-know-what-they-dont-know-internal-states-mainly-reflect-knowledge-recall-rather-than-truthfulness] Cluster-separability metrics (Silhouette, Davies-Bouldin on t-SNE) show knowledge-recall and truthfulness occupy measurably different regions, replicated on Mistral-7B-v0.3 [cheang-etal-2026-do-llms-really-know-what-they-dont-know-internal-states-mainly-reflect-knowledge-recall-rather-than-truthfulness]

models: Llama-3-8B · method: Linear probing
Hallucination Basins: A Dynamic Framework for Understanding and Controlling LLM Hallucinations (2026)measured

A diff-of-means hallucination direction causally raises hallucination dose-dependently

Details

Cherukuri & Varshney formalize hallucinations as basin attractors across Llama-3.2-1B/3B, Gemma-2-2B, Qwen2.5-1.5B, Llama-3.1-8B, and Mistral-7B-v0.3 [cherukuri-varshney-2026-hallucination-basins-a-dynamic-framework-for-understanding-and-controlling-llm-hallucinations] Basin separation is strongly task-dependent: factoid tasks show sharp centroid separation (variance ratio up to 4.55, AUROC up to 1.000 on MuSiQue) while summarization and TruthfulQA hover near chance [cherukuri-varshney-2026-hallucination-basins-a-dynamic-framework-for-understanding-and-controlling-llm-hallucinations] Interpolating factual hidden states toward the hallucination centroid via a diff-of-means steering vector produces a monotonic dose-response increase in hallucination probability, exceeding random and orthogonal controls [cherukuri-varshney-2026-hallucination-basins-a-dynamic-framework-for-understanding-and-controlling-llm-hallucinations]

models: Llama-3.2-1B, Llama-3.2-3B, Llama-3.1-8B · method: Difference-in-means direction, Activation Steering (Addition)
How Language Directions Align with Token Geometry in Multilingual LLMs (2026)measured

Language identity is linearly separable from layer 1; alignment tracks pretraining mix

Details

Kim & Lee probe all layers of six multilingual LLMs across five XNLI languages, training linear and MLP probes to classify language identity [kim-lee-2026-language-directions-token-geometry-multilingual-llms] Language separability jumps sharply in the first transformer block (+76.4 points) and stays almost fully linearly separable throughout depth (linear-probe accuracy 99.8%, only 0.58 points below the MLP) [kim-lee-2026-language-directions-token-geometry-multilingual-llms] A Token-Language Alignment metric tracks pretraining composition: Chinese-inclusive models reach 16.43% Chinese Match@Peak versus 3.90% for English-centric ones (4.21x) [kim-lee-2026-language-directions-token-geometry-multilingual-llms] Latin-script languages show uniformly low alignment regardless of model, confounded by shared script; no causal intervention is performed [kim-lee-2026-language-directions-token-geometry-multilingual-llms]

models: Llama-3.1-8B, OpenMath2-Llama3.1-8B · method: Linear probing
Layerwise Recall and the Geometry of Interwoven Knowledge in LLMs (2025)measured

A PCA spiral in Llama hidden states aligns with the periodic table

Details

Lei & Cooper apply PCA to Llama-3-8B and Llama-3.1-70B hidden states on chemistry prompts, finding a 3D spiral in the top components that tracks the periodic table's atomic-number ordering [lei-cooper-2025-layerwise-recall-and-the-geometry-of-interwoven-knowledge-in-llms] Per-property spiral-fit consistency scores reach up to about 0.90 for some element properties and lower (around 0.65) for others [lei-cooper-2025-layerwise-recall-and-the-geometry-of-interwoven-knowledge-in-llms] Layer-wise linear probing shows middle layers encode element attributes continuously and overlapping while deeper layers sharpen them into more categorically separated clusters [lei-cooper-2025-layerwise-recall-and-the-geometry-of-interwoven-knowledge-in-llms]

models: Llama-3-8B, Llama 3.1 70B · method: Linear probing
Finding Lexical Identity and Inflectional Morphology in Modern Language Models (2025)measured

Morphology stays linearly separable while lexical identity turns nonlinear with depth

Details

Li & Subramani train ridge-regression linear probes and nonlinear MLP probes across 25 pretrained models to predict word lemma and inflectional features [li-subramani-2025-finding-lexical-identity-and-inflectional-morphology-in-modern-language-models] Lemma is linearly separable in early layers (0.8-1.0) but linear accuracy declines toward final layers while MLP accuracy declines much less, so lexical identity becomes increasingly nonlinearly encoded [li-subramani-2025-finding-lexical-identity-and-inflectional-morphology-in-modern-language-models] Inflectional morphology stays linearly separable at 0.9-1.0 uniformly across every layer, with MLP probes adding no benefit [li-subramani-2025-finding-lexical-identity-and-inflectional-morphology-in-modern-language-models] Control-task selectivity confirms inflection is a generalizable linear abstraction (selectivity 0.4-0.6) while lemma decodability reflects near-zero-selectivity memorization [li-subramani-2025-finding-lexical-identity-and-inflectional-morphology-in-modern-language-models]

models: Llama-3.1-8B · method: Linear probing, Mlp probing
What Really Controls Temporal Reasoning in LLMs: Tokenisation or Representation of Time? (2026)measured

Temporal-probe decodability predicts reasoning in high-resource languages

Details

Bhatia et al. introduce MultiTempBench and evaluate 20 LLMs, combining a tokenization-fragmentation metric with linear-regression probes decoding Year/Month/Day (probe R^2 as temporal linearity) [bhatia-etal-2026-what-really-controls-temporal-reasoning-in-llms] A crossed mixed-effects regression over 285,000 predictions finds temporal linearity is the strongest predictor of reasoning accuracy in high-resource languages (English r=0.77) [bhatia-etal-2026-what-really-controls-temporal-reasoning-in-llms] Tokenization fragmentation dominates in low-resource languages (Hausa r=-0.97); the finding is purely correlational [bhatia-etal-2026-what-really-controls-temporal-reasoning-in-llms]

models: Llama-3.1-8B, Llama-2-7B · method: Linear probing
LLM Agents Already Know When to Call Tools — Even Without Reasoning (2026)measured

A probe predicts tool-call necessity even when generation fails to

Details

Sun et al. show prompting and Reason-then-Act give unreliable control over tool-call decisions, sometimes collapsing accuracy (Llama-3.1-8B-Instruct 79.5% to 31.2%) [sun-etal-2026-llm-agents-already-know-when-to-call-tools-even-without-reasoning] A logistic-regression probe on pre-generation all-layer last-token hidden states predicts binary tool necessity at AUROC 0.89-0.96 across six LLMs [sun-etal-2026-llm-agents-already-know-when-to-call-tools-even-without-reasoning] This includes the two Llama models whose own generation fails to express the knowledge, dissociating what a model knows internally from what it outputs [sun-etal-2026-llm-agents-already-know-when-to-call-tools-even-without-reasoning]

models: Llama-3.1-8B-Instruct, Llama 3.3 70B Instruct · method: Linear probing
Unlocking the Future: Look-Ahead Planning Mechanistic Interpretability in LLMs (2024)measured

Linear probes decode future Blocksworld planning steps from one pass

Details

Men et al. train linear probes (versus a nonlinear control) on hidden states of Llama-2-7b-chat and Vicuna-7B fine-tuned on Blocksworld planning [men-etal-2024-unlocking-the-future-look-ahead-planning-mechanistic-interpretability-in-llms] A linear probe predicts the 6th-step-ahead decision from the 1st-step representation at 0.51 accuracy, decaying smoothly with prediction distance [men-etal-2024-unlocking-the-future-look-ahead-planning-mechanistic-interpretability-in-llms] Linear and nonlinear probes track the same decay, evidence the look-ahead information is linearly encoded; MHSA key-masking confirms which attention paths carry it [men-etal-2024-unlocking-the-future-look-ahead-planning-mechanistic-interpretability-in-llms]

models: Llama-2-7B-Chat · method: Linear probing, Causal interventions (steering)
Catching Rationalization in the Act: Detecting Motivated Reasoning Before and After CoT via Activation Probing (2026)measured

Probes detect motivated reasoning before and after chain-of-thought

Details

Mirtaheri & Belkin use a paired hinted/unhinted framework to label motivated reasoning across Qwen3-8B, Llama-3.1-8B-Instruct, and Gemma-3-4B, probing activations with recursive feature machines [mirtaheri-belkin-2026-catching-rationalization-in-the-act-detecting-motivated-reasoning-before-and-after-cot-via-activation-probing] Pre-generation probes match a GPT-5-nano full-trace CoT monitor and post-generation probes outperform it [mirtaheri-belkin-2026-catching-rationalization-in-the-act-detecting-motivated-reasoning-before-and-after-cot-via-activation-probing] A hint-recovery probe shows a U-shaped accuracy curve across CoT tokens, so the model internally re-engages the hinted answer even when its CoT never mentions it [mirtaheri-belkin-2026-catching-rationalization-in-the-act-detecting-motivated-reasoning-before-and-after-cot-via-activation-probing]

models: Llama-3.1-8B-Instruct · method: Recursive Feature Machine (RFM)
Analysing the Residual Stream of Language Models Under Knowledge Conflicts (2024)measured

Residual-stream probes detect knowledge conflict and reliance

Details

- Logistic-regression probes on the mid-layer residual stream detect parametric-vs-contextual knowledge conflict with ~90% accuracy (peaking around layer 14 of Llama-3-8B on NQSwap). [zhao-etal-2024-analysing-the-residual-stream-of-language-models-under-knowledge-conflicts] - The same probes predict which knowledge source the model will rely on before generation (peaking around layers 16-17). [zhao-etal-2024-analysing-the-residual-stream-of-language-models-under-knowledge-conflicts] - When the model relies on contextual knowledge the residual stream is distinctly more skewed than for parametric knowledge (most pronounced in layers ~20-30). [zhao-etal-2024-analysing-the-residual-stream-of-language-models-under-knowledge-conflicts] - Tested on base Llama-3-8B and Llama-2-7B over NQSwap, Macnoise and ConflictQA; observational. [zhao-etal-2024-analysing-the-residual-stream-of-language-models-under-knowledge-conflicts]

models: Llama-3-8B, Llama-2-7B · method: Linear probing
Tracing Relational Knowledge Recall in Large Language Models (2026)measured

Per-head attention-contribution probes decode relation type above full-state probes

Details

Popovic & Farber decompose each attention head's contribution to the residual stream via DLA-style attribution and probe these per-head features for relation type [popovic-farber-2026-tracing-relational-knowledge-recall-in-large-language-models] Linear probes decode FewRel relation type in a 5-way-5-shot task at 89.66-91.09% accuracy across Llama-3.2-1B/3B-Instruct, Llama-3.1-8B-Instruct, and Qwen3-4B-Instruct [popovic-farber-2026-tracing-relational-knowledge-recall-in-large-language-models] This exceeds probes fit on the full attention-state baseline (75.06-86.79%) on the same models and task [popovic-farber-2026-tracing-relational-knowledge-recall-in-large-language-models]

models: Llama-3.2-1B-Instruct, Llama-3.2-3B-Instruct, Llama-3.1-8B-Instruct · method: Linear probing, Direct Logit Attribution (DLA)
Reasoning Models Know When They're Right: Probing Hidden States for Self-Verification (2025)measured

A probe reads answer correctness before reasoning models state it

Details

Zhang et al. fit probes (many converging to purely linear) on hidden states at intermediate-answer positions across six reasoning models [zhang-etal-2025-reasoning-models-know-when-theyre-right-probing-hidden-states-for-self-verification] Correctness is predicted at ROC-AUC above 0.7 with calibration error under 0.1 in every model, including for future not-yet-stated answers later in the trace [zhang-etal-2025-reasoning-models-know-when-theyre-right-probing-hidden-states-for-self-verification] The effect is much weaker in a non-reasoning baseline (Llama-3.1-8B-Instruct), tying the signal to reasoning-specific training [zhang-etal-2025-reasoning-models-know-when-theyre-right-probing-hidden-states-for-self-verification] Used as an early-exit verifier, the probe cuts inference tokens by 24% with no accuracy loss [zhang-etal-2025-reasoning-models-know-when-theyre-right-probing-hidden-states-for-self-verification]

models: Llama-3.1-8B-Instruct · method: Linear probing, Mlp probing
Unifying Attention Heads and Task Vectors via Hidden State Geometry in In-Context Learning (2025)measured

ICL separability rises first, then alignment with the unembedding

Details

Yang et al. bound classification accuracy by hidden-state linear separability, requiring both output alignment and directional alignment with the label unembedding-difference vector [yang-etal-2025] Across 7 models on 6 datasets, ICL proceeds in two stages: separability rises rapidly in early layers, then middle-to-late layers spike four alignment measures together [yang-etal-2025] The ICL-versus-zero-shot separability gap is small despite an ~80-point accuracy gap, so ICL's gains come almost entirely from improved alignment, not separability [yang-etal-2025] Ablating Previous-Token Heads collapses separability while sparing alignment, and ablating Induction Heads collapses alignment and accuracy (to 2.5%), giving a geometric account of function vectors [yang-etal-2025]

models: Llama-2-7B, Llama-2-13B, Llama-2-70B, Llama-3-8B, Llama-3-70B · method: Linear probing, PCA, Causal interventions (steering), Function vector extraction (causal-mediation head selection + sum), Separability/alignment decomposition
LLM Reasoning as Trajectories: Step-Specific Representation Geometry and Correctness Signals (2026)measured

Chain-of-thought traverses step-specific subspaces that sharpen with depth

Details

Sun et al. train per-layer per-step one-vs-rest logistic probes on activations preceding each reasoning-step marker in Llama-3.1-8B Base, Instruct, and R1-Distill [sun-etal-2026-llm-reasoning-as-trajectories] Later-step separability starts low at the embedding layer and exceeds 0.90 only after several layers, while early steps are separable from layer 0; shuffling controls collapse to about 0.59 [sun-etal-2026-llm-reasoning-as-trajectories] Probes transfer across training regimes, prompt formats, and datasets (0.83-1.00), and late-trajectory PCA features predict eventual correctness mid-generation (ROC-AUC up to 0.87) [sun-etal-2026-llm-reasoning-as-trajectories] A termination-direction steering vector causally controls reasoning length, and a rank-32 trajectory-correction update raises 6-7-step GSM8K accuracy by about 7.6 points while preserving correct answers [sun-etal-2026-llm-reasoning-as-trajectories]

models: Llama-3.1-8B, Llama-3.1-8B-Instruct · method: Linear probing, Causal interventions (steering)
The Obfuscation Atlas: Mapping Where Honesty Emerges in RLVR with Deception Probes (2026)measured

Layerwise honesty-decodability probes drift sharply off-domain under real RLVR training

Details

Taufeeque, Heimersheim, Gleave & Cundy train per-layer logistic-regression probes mapping where honesty/deception is linearly decodable in real Llama-3-8B-Instruct, Gemma-3-12B/27B-IT, and Llama-3-70B-Instruct, then run real LoRA+GRPO RLVR training on an MBPP-Honeypot coding environment where reward hacking naturally emerges [taufeeque-etal-2026-obfuscation-atlas-honesty-emerges-in-rlvr] Pre-RL probe AUC is near-ceiling (0.991-1.000); after RLVR training with no detector penalty, AUC on off-domain deception probes collapses (e.g. 0.991 to 0.376, 0.996 to 0.655) while on-domain MBPP-probe AUC stays at 1.000, indicating representation drift rather than deliberate evasion [taufeeque-etal-2026-obfuscation-atlas-honesty-emerges-in-rlvr] Adding an explicit detector penalty during training instead produces genuinely evasive deceptive behavior in 96.4 percent of seeds (27 of 28) [taufeeque-etal-2026-obfuscation-atlas-honesty-emerges-in-rlvr]

models: Llama-3-8B-Instruct · method:
The Blessing and Curse of Dimensionality in Safety Alignment (2025)measured

Linear safety separability emerges above a hidden-dimension threshold

Details

Teo, Abdullaev & Nguyen show via PCA across Qwen0.5B, GPT2-XL, Qwen7B, and Llama2-7B that emotion clusters become distinctly separable only above ~2,000-3,000 hidden dimensions [teo-etal-2025-blessing-and-curse-of-dimensionality-in-safety-alignment] This emergent linear separability is exactly what white-box jailbreaks exploit, collapsing Llama2-7B-Chat's refusal/safety from 0.97/0.99 to 0.03/0.14 [teo-etal-2025-blessing-and-curse-of-dimensionality-in-safety-alignment] Two dimension-reduction defenses (FJLT random projection and a learned bottleneck autoencoder) restore refusal/safety to 0.94-0.96, with the bottleneck retaining about 70% of benign helpfulness [teo-etal-2025-blessing-and-curse-of-dimensionality-in-safety-alignment] A Rademacher-complexity bound explains why shrinking the ambient dimension raises the sample cost of finding a steering direction [teo-etal-2025-blessing-and-curse-of-dimensionality-in-safety-alignment]

models: Llama-2-7B, Llama-2-7B-Chat · method: Difference-in-means direction, Subspace-Compression Defense (FJLT / Bottleneck Autoencoder)
Representational Depth of Evaluation Awareness Shifts With Scale in Open-Weight LMs (2026)measured

Evaluation-awareness probe depth shifts late-to-early with model scale

Details

Manek projects a diff-of-means evaluation-awareness direction (from 203 contrastive prompt pairs) onto residual-stream activations at every layer across 11 open-weight models (Qwen2.5 0.5B-32B, Gemma2 2B/9B/27B, Llama-3.2 1B/3B) [manek-2026-representational-depth-of-evaluation-awareness-shifts-with-scale] The relative depth at which per-layer decoding AUROC peaks shifts from late layers in small models to the earliest layers in large ones (Qwen2.5 1.5B/3B peak at 0.96-0.97 vs 14B/32B at 0.021-0.031) [manek-2026-representational-depth-of-evaluation-awareness-shifts-with-scale] Gemma2 shows the same late-to-early shift (0.885 to 0.304) while Llama-3.2 stays mid-layer across both tested sizes [manek-2026-representational-depth-of-evaluation-awareness-shifts-with-scale] Peak AUROC ranges 0.586-0.873 and is non-monotonic with scale within a family [manek-2026-representational-depth-of-evaluation-awareness-shifts-with-scale]

models: Llama-3.2-1B, Llama-3.2-3B · method: Direction Extraction, Linear probing, Geometric analysis
Large Language Models Encode Semantics and Alignment in Linearly Separable Representations (2025)measured

Topic subspaces sharpen with depth; a centroid direction induces CoT

Details

Saglam et al. study 11 autoregressive models on six arXiv topics, measuring per-layer hard-margin SVM separability and PCA intrinsic dimensionality [saglam-etal-2025-linear-subspaces] Under 10% of principal components explain nearly all variance, so topics occupy a near-affine low-dimensional subspace; separability rises toward final layers, reaching 100% SVM accuracy on all topic pairs in the largest models [saglam-etal-2025-linear-subspaces] A chain-of-thought framing distinction becomes linearly separable even more sharply than topic identity [saglam-etal-2025-linear-subspaces] Adding the CoT-vs-non-CoT centroid-difference direction to the hidden state reliably induces CoT-style responses (flagged preliminary), and a lightweight MLP guardrail built on the finding halves harmful responses [saglam-etal-2025-linear-subspaces]

models: Llama-3.1-8B, Llama-3.1-8B-Instruct, Llama-3.2-3B · method: Linear probing, PCA, Activation Steering (Addition)
Multilingual Language Models Encode Script Over Linguistic Structure (2026)measured

Language units are script-conditioned; typology decodes with depth

Details

Verma et al. use LAPE on MLP neurons and SAE-LAPE on latent features in Llama-3.2-1B and Gemma-2-2B to ask whether language units encode identity or surface script [verma-etal-2026-multilingual-lms-encode-script-over-linguistic-structure] Romanizing a non-Latin language produces a unit set nearly disjoint from both the native-script and English sets (Jaccard <0.3), a script-conditioned third subspace, while word-order shuffling leaves most raw-neuron units intact [verma-etal-2026-multilingual-lms-encode-script-over-linguistic-structure] Linear probing against lang2vec shows script-invariant units carry the strongest typological signal, and typological accessibility is depth-dependent (genealogy early, phonology deepest) [verma-etal-2026-multilingual-lms-encode-script-over-linguistic-structure] Causal ablation shows perplexity is most disrupted when script-invariant or order-invariant units are ablated, so functional necessity tracks surface-invariance rather than typological alignment [verma-etal-2026-multilingual-lms-encode-script-over-linguistic-structure]

models: Llama-3.2-1B · method: Language Activation Probability Entropy (LAPE), Sparse Autoencoders (SAE), Causal interventions (steering)
Two Axes of LLM Abstention: Answer Correctness and Question Answerability (2026)measured

A probe detects unanswerability far better than confidence, not shrinking with scale

Details

Wagner fits logistic-regression probes on final-prompt-token hidden states of Gemma 2 2B-it, Qwen2.5-3B/7B/14B, and Llama-3.1-8B [wagner-2026-two-axes-of-llm-abstention-answer-correctness-and-question-answerability] Hidden-state answerability readout reaches AUROC 0.97-0.99 across all five models versus only 0.54-0.67 for output-confidence readouts, a gap that does not shrink with scale [wagner-2026-two-axes-of-llm-abstention-answer-correctness-and-question-answerability] On naturally-occurring false-premise questions all confidence signals stay near chance while the probe reaches 0.69-0.77 AUROC [wagner-2026-two-axes-of-llm-abstention-answer-correctness-and-question-answerability] Routing a premise-checking instruction through the probe roughly triples challenge precision; no causal steering is performed [wagner-2026-two-axes-of-llm-abstention-answer-correctness-and-question-answerability]

models: Llama-3.1-8B · method: Linear probing
Monitoring Emergent Reward Hacking During Generation via Internal Activations (2026)measured

A probe on SAE-PCA activations detects reward hacking token-by-token

Details

Wilhelm, Wittkopp & Kao train a per-layer sparse autoencoder on the last four layers of Qwen2.5-Instruct-7B, LLaMA-3.1-8B, and Falcon3-7B, then fit a logistic-regression probe on PCA-reduced SAE features for a per-token hack probability [wilhelm-wittkopp-kao-2026-monitoring-emergent-reward-hacking-during-generation-via-internal-activations] Against a GPT-4o judge (metric F1), the probe reaches F1=1.000 only on the benign control-adapter data, while reward-hacking detection F1 is 0.760-0.961 (e.g. Llama 0.961, Qwen 0.784) [wilhelm-wittkopp-kao-2026-monitoring-emergent-reward-hacking-during-generation-via-internal-activations] Reward-hacking versus benign activations are linearly separable in the SAE+PCA-reduced space; no causal steering is performed [wilhelm-wittkopp-kao-2026-monitoring-emergent-reward-hacking-during-generation-via-internal-activations]

models: Llama-3.1-8B · method: Sparse Autoencoders (SAE), PCA, Linear probing
Decoding Emotion in the Deep: A Systematic Study of How LLMs Represent, Retain, and Express Emotion (2025)measured

Emotion clusters peak at mid-depth and tighten with scale

Details

Zhang & Zhong train two-layer MLP probes on frozen hidden states of Qwen3 (0.6B-8B) and LLaMA-3 (1B-8B), finding 7-class emotion clusters tight and well-separated for larger models but diffuse for smaller [zhang-zhong-2025-decoding-emotion-in-the-deep] Layer-wise probe accuracy rises from chance (0.143) to a peak of 0.78-0.80 at 50-75% relative depth rather than the final layer [zhang-zhong-2025-decoding-emotion-in-the-deep] An offset-aware probe shows emotion-specific signal decaying differentially across up to 400 generated tokens [zhang-zhong-2025-decoding-emotion-in-the-deep]

models: Llama-3.2-1B, Llama-3.2-3B, Llama-3.1-8B · method: Mlp probing, PCA

Qwen

Understanding Moral Reasoning Trajectories in LLMs: Toward Probing-Based Explainability (2026)measured

A moral-framework probe decodes and steers mid-reasoning ethics

Details

Huang, Kwak & An train linear probes on last-token residual-stream states across all 81 layers of Llama-3.3-70B and Qwen2.5-72B to predict a 5-dimensional moral-framework distribution [huang-etal-2026-understanding-moral-reasoning-trajectories-in-llms] Framework encoding localizes to model-specific layers (63/81 for Llama-3.3-70B, 17/81 for Qwen2.5-72B), achieving 13.8-22.6% lower KL divergence than a training-prior baseline [huang-etal-2026-understanding-moral-reasoning-trajectories-in-llms] Steering along the probe direction modestly reduces the Framework Drift Rate (6.7% at Llama layer 6; 8.9% at Qwen layer 1) [huang-etal-2026-understanding-moral-reasoning-trajectories-in-llms] Probing and steering are restricted to the two open-weight models, while closed models (GPT-5, GPT-4o family) are evaluated only behaviorally [huang-etal-2026-understanding-moral-reasoning-trajectories-in-llms]

models: Qwen2.5-72B · method: Linear probing, Causal interventions (steering)
What LLM Forecasters Know but Don't Say: Probing Internal Representations for Calibration and Faithfulness (2026)measured

A covariance-pooling probe beats verbalized confidence and catches stealth belief shifts

Details

Sarfati et al. train pooling probes on frozen residual-stream activations of Eternis-Forecaster-8B, GLM-4.5-Air, and Qwen3-8B to read out forecast correctness and confidence [sarfati-etal-2026-what-llm-forecasters-know-but-dont-say-probing-internal-representations-for-calibration-and-faithfulness] A layer-21 covariance probe halves calibration error (ECE 0.044 vs 0.093 for verbalized confidence at matched AUROC ~0.756) and holds out-of-distribution where verbalized AUROC collapses to 0.587 [sarfati-etal-2026-what-llm-forecasters-know-but-dont-say-probing-internal-representations-for-calibration-and-faithfulness] The probe's activation shift under evidence ablation/injection tracks behavioral change at Spearman rho=0.565 (vs 0.215 for reasoning-text changes), predicting the direction in 83.6% of cases including 107 stealth cases where the CoT shows no change [sarfati-etal-2026-what-llm-forecasters-know-but-dont-say-probing-internal-representations-for-calibration-and-faithfulness]

models: Qwen3-8B · method: Linear probing
Tool-Call Dependency Structure Is Linearly Decodable in LLM Agent Residual Streams (2026)measured

A probe decodes the tool-call dependency DAG that causally propagates

Details

Sun & Kazakov train a logistic-regression edge probe on all 65 residual-stream layers of Qwen3-32B to recover the tool-call dependency DAG in tau-bench's retail split [sun-kazakov-2026-tool-call-dependency-structure-is-linearly-decodable-in-llm-agent-residual-streams] The probe reaches AUROC 0.869 on held-out pairs, above a random-label control (0.491), a positional-only baseline (0.792), a random-init model (0.738), and an n-gram decoder (0.830) [sun-kazakov-2026-tool-call-dependency-structure-is-linearly-decodable-in-llm-agent-residual-streams] Signal emerges sharply by layer 14 (~22% depth) and plateaus through the final layer [sun-kazakov-2026-tool-call-dependency-structure-is-linearly-decodable-in-llm-agent-residual-streams] Per-layer activation patching shifts the probe's prediction at later non-patched layers toward a donor trajectory, so the representation causally propagates [sun-kazakov-2026-tool-call-dependency-structure-is-linearly-decodable-in-llm-agent-residual-streams]

models: Qwen3-32B · method: Linear probing, Activation patching
Proxy Reward Internalization and Mechanistic Exploitation: A Learned Precursor to Reward Hacking and Its Generalization (2026)measured

Diff-in-means reward-hacking directions forecast and causally control misalignment

Details

Beigi, Jin & Huang extract three diff-in-means directions (context-sycophantic-agreement, premise-rejection, evaluator-rejection) during RL training of Qwen2.5-Coder-14B-Instruct and track their projection score across checkpoints [beigi-jin-huang-2026-proxy-reward-internalization-and-mechanistic-exploitation] Each direction is linearly decodable (AUROC 0.85-0.90) and a linear fit from probe score forecasts future out-of-domain misalignment (R^2=0.77), preceding the misalignment rise by roughly 45 training steps [beigi-jin-huang-2026-proxy-reward-internalization-and-mechanistic-exploitation] Joint ablation of the three directions lowers the hack rate by 26 points while preserving coding accuracy, and injection along them increases hacking [beigi-jin-huang-2026-proxy-reward-internalization-and-mechanistic-exploitation] The probe methodology generalizes to Qwen2.5-Coder 1.5B/7B/32B, Llama-3.1-8B-Instruct, and OLMo-7B [beigi-jin-huang-2026-proxy-reward-internalization-and-mechanistic-exploitation]

models: Qwen2.5-Coder-14B-Instruct · method: Difference-in-means direction
Hallucination Basins: A Dynamic Framework for Understanding and Controlling LLM Hallucinations (2026)measured

A diff-of-means hallucination direction causally raises hallucination dose-dependently

Details

Cherukuri & Varshney formalize hallucinations as basin attractors across Llama-3.2-1B/3B, Gemma-2-2B, Qwen2.5-1.5B, Llama-3.1-8B, and Mistral-7B-v0.3 [cherukuri-varshney-2026-hallucination-basins-a-dynamic-framework-for-understanding-and-controlling-llm-hallucinations] Basin separation is strongly task-dependent: factoid tasks show sharp centroid separation (variance ratio up to 4.55, AUROC up to 1.000 on MuSiQue) while summarization and TruthfulQA hover near chance [cherukuri-varshney-2026-hallucination-basins-a-dynamic-framework-for-understanding-and-controlling-llm-hallucinations] Interpolating factual hidden states toward the hallucination centroid via a diff-of-means steering vector produces a monotonic dose-response increase in hallucination probability, exceeding random and orthogonal controls [cherukuri-varshney-2026-hallucination-basins-a-dynamic-framework-for-understanding-and-controlling-llm-hallucinations]

models: Qwen2.5-1.5B · method: Difference-in-means direction, Activation Steering (Addition)
How Language Directions Align with Token Geometry in Multilingual LLMs (2026)measured

Language identity is linearly separable from layer 1; alignment tracks pretraining mix

Details

Kim & Lee probe all layers of six multilingual LLMs across five XNLI languages, training linear and MLP probes to classify language identity [kim-lee-2026-language-directions-token-geometry-multilingual-llms] Language separability jumps sharply in the first transformer block (+76.4 points) and stays almost fully linearly separable throughout depth (linear-probe accuracy 99.8%, only 0.58 points below the MLP) [kim-lee-2026-language-directions-token-geometry-multilingual-llms] A Token-Language Alignment metric tracks pretraining composition: Chinese-inclusive models reach 16.43% Chinese Match@Peak versus 3.90% for English-centric ones (4.21x) [kim-lee-2026-language-directions-token-geometry-multilingual-llms] Latin-script languages show uniformly low alignment regardless of model, confounded by shared script; no causal intervention is performed [kim-lee-2026-language-directions-token-geometry-multilingual-llms]

models: Qwen-2.5-7B, Qwen2.5-Math-7B, OpenR1-Qwen-7B · method: Linear probing
Encoded but Not Routed: Explaining the Table-Chart Gap in Scientific Claim Verification (2026)measured

VLMs encode chart signal linearly but do not route it to decisions

Details

Kumar et al. train layer-wise logistic-regression probes on hidden states of Qwen2.5-VL-7B/32B-Instruct and InternVL3-8B for chart-claim verification [kumar-etal-2026-encoded-but-not-routed-explaining-the-table-chart-gap-in-scientific-claim-verification] On InternVL3-8B the mean-pool probe reaches 79.0% accuracy while the model's own decision reaches only 56.2%; on Qwen2.5-VL-32B probe AUROC is 88.9% versus 71.9% at the last token (McNemar p<0.01) [kumar-etal-2026-encoded-but-not-routed-explaining-the-table-chart-gap-in-scientific-claim-verification] An attention analysis finds the Qwen models attend to image tokens at only 4-11% of a proportional baseline at the final layer, while InternVL3 attends at about 93% [kumar-etal-2026-encoded-but-not-routed-explaining-the-table-chart-gap-in-scientific-claim-verification] The authors note the analysis is diagnostic, not causal [kumar-etal-2026-encoded-but-not-routed-explaining-the-table-chart-gap-in-scientific-claim-verification]

models: Qwen2.5-VL 7B Instruct, Qwen2.5-VL 32B Instruct · method: Linear probing
Finding Lexical Identity and Inflectional Morphology in Modern Language Models (2025)measured

Morphology stays linearly separable while lexical identity turns nonlinear with depth

Details

Li & Subramani train ridge-regression linear probes and nonlinear MLP probes across 25 pretrained models to predict word lemma and inflectional features [li-subramani-2025-finding-lexical-identity-and-inflectional-morphology-in-modern-language-models] Lemma is linearly separable in early layers (0.8-1.0) but linear accuracy declines toward final layers while MLP accuracy declines much less, so lexical identity becomes increasingly nonlinearly encoded [li-subramani-2025-finding-lexical-identity-and-inflectional-morphology-in-modern-language-models] Inflectional morphology stays linearly separable at 0.9-1.0 uniformly across every layer, with MLP probes adding no benefit [li-subramani-2025-finding-lexical-identity-and-inflectional-morphology-in-modern-language-models] Control-task selectivity confirms inflection is a generalizable linear abstraction (selectivity 0.4-0.6) while lemma decodability reflects near-zero-selectivity memorization [li-subramani-2025-finding-lexical-identity-and-inflectional-morphology-in-modern-language-models]

models: Qwen2.5-1.5B-Instruct · method: Linear probing, Mlp probing
What Really Controls Temporal Reasoning in LLMs: Tokenisation or Representation of Time? (2026)measured

Temporal-probe decodability predicts reasoning in high-resource languages

Details

Bhatia et al. introduce MultiTempBench and evaluate 20 LLMs, combining a tokenization-fragmentation metric with linear-regression probes decoding Year/Month/Day (probe R^2 as temporal linearity) [bhatia-etal-2026-what-really-controls-temporal-reasoning-in-llms] A crossed mixed-effects regression over 285,000 predictions finds temporal linearity is the strongest predictor of reasoning accuracy in high-resource languages (English r=0.77) [bhatia-etal-2026-what-really-controls-temporal-reasoning-in-llms] Tokenization fragmentation dominates in low-resource languages (Hausa r=-0.97); the finding is purely correlational [bhatia-etal-2026-what-really-controls-temporal-reasoning-in-llms]

models: Qwen3-0.6B, Qwen3-4B, Qwen3-14B · method: Linear probing
LLM Agents Already Know When to Call Tools — Even Without Reasoning (2026)measured

A probe predicts tool-call necessity even when generation fails to

Details

Sun et al. show prompting and Reason-then-Act give unreliable control over tool-call decisions, sometimes collapsing accuracy (Llama-3.1-8B-Instruct 79.5% to 31.2%) [sun-etal-2026-llm-agents-already-know-when-to-call-tools-even-without-reasoning] A logistic-regression probe on pre-generation all-layer last-token hidden states predicts binary tool necessity at AUROC 0.89-0.96 across six LLMs [sun-etal-2026-llm-agents-already-know-when-to-call-tools-even-without-reasoning] This includes the two Llama models whose own generation fails to express the knowledge, dissociating what a model knows internally from what it outputs [sun-etal-2026-llm-agents-already-know-when-to-call-tools-even-without-reasoning]

models: Qwen3-1.7B, Qwen3-4B-Instruct, Qwen3-14B, Qwen3-32B · method: Linear probing
Catching Rationalization in the Act: Detecting Motivated Reasoning Before and After CoT via Activation Probing (2026)measured

Probes detect motivated reasoning before and after chain-of-thought

Details

Mirtaheri & Belkin use a paired hinted/unhinted framework to label motivated reasoning across Qwen3-8B, Llama-3.1-8B-Instruct, and Gemma-3-4B, probing activations with recursive feature machines [mirtaheri-belkin-2026-catching-rationalization-in-the-act-detecting-motivated-reasoning-before-and-after-cot-via-activation-probing] Pre-generation probes match a GPT-5-nano full-trace CoT monitor and post-generation probes outperform it [mirtaheri-belkin-2026-catching-rationalization-in-the-act-detecting-motivated-reasoning-before-and-after-cot-via-activation-probing] A hint-recovery probe shows a U-shaped accuracy curve across CoT tokens, so the model internally re-engages the hinted answer even when its CoT never mentions it [mirtaheri-belkin-2026-catching-rationalization-in-the-act-detecting-motivated-reasoning-before-and-after-cot-via-activation-probing]

models: Qwen3-8B · method: Recursive Feature Machine (RFM)
Tracing Relational Knowledge Recall in Large Language Models (2026)measured

Per-head attention-contribution probes decode relation type above full-state probes

Details

Popovic & Farber decompose each attention head's contribution to the residual stream via DLA-style attribution and probe these per-head features for relation type [popovic-farber-2026-tracing-relational-knowledge-recall-in-large-language-models] Linear probes decode FewRel relation type in a 5-way-5-shot task at 89.66-91.09% accuracy across Llama-3.2-1B/3B-Instruct, Llama-3.1-8B-Instruct, and Qwen3-4B-Instruct [popovic-farber-2026-tracing-relational-knowledge-recall-in-large-language-models] This exceeds probes fit on the full attention-state baseline (75.06-86.79%) on the same models and task [popovic-farber-2026-tracing-relational-knowledge-recall-in-large-language-models]

models: Qwen3-4B-Instruct · method: Linear probing, Direct Logit Attribution (DLA)
Reasoning Models Know When They're Right: Probing Hidden States for Self-Verification (2025)measured

A probe reads answer correctness before reasoning models state it

Details

Zhang et al. fit probes (many converging to purely linear) on hidden states at intermediate-answer positions across six reasoning models [zhang-etal-2025-reasoning-models-know-when-theyre-right-probing-hidden-states-for-self-verification] Correctness is predicted at ROC-AUC above 0.7 with calibration error under 0.1 in every model, including for future not-yet-stated answers later in the trace [zhang-etal-2025-reasoning-models-know-when-theyre-right-probing-hidden-states-for-self-verification] The effect is much weaker in a non-reasoning baseline (Llama-3.1-8B-Instruct), tying the signal to reasoning-specific training [zhang-etal-2025-reasoning-models-know-when-theyre-right-probing-hidden-states-for-self-verification] Used as an early-exit verifier, the probe cuts inference tokens by 24% with no accuracy loss [zhang-etal-2025-reasoning-models-know-when-theyre-right-probing-hidden-states-for-self-verification]

models: QwQ-32B · method: Linear probing, Mlp probing
The Blessing and Curse of Dimensionality in Safety Alignment (2025)measured

Linear safety separability emerges above a hidden-dimension threshold

Details

Teo, Abdullaev & Nguyen show via PCA across Qwen0.5B, GPT2-XL, Qwen7B, and Llama2-7B that emotion clusters become distinctly separable only above ~2,000-3,000 hidden dimensions [teo-etal-2025-blessing-and-curse-of-dimensionality-in-safety-alignment] This emergent linear separability is exactly what white-box jailbreaks exploit, collapsing Llama2-7B-Chat's refusal/safety from 0.97/0.99 to 0.03/0.14 [teo-etal-2025-blessing-and-curse-of-dimensionality-in-safety-alignment] Two dimension-reduction defenses (FJLT random projection and a learned bottleneck autoencoder) restore refusal/safety to 0.94-0.96, with the bottleneck retaining about 70% of benign helpfulness [teo-etal-2025-blessing-and-curse-of-dimensionality-in-safety-alignment] A Rademacher-complexity bound explains why shrinking the ambient dimension raises the sample cost of finding a steering direction [teo-etal-2025-blessing-and-curse-of-dimensionality-in-safety-alignment]

models: Qwen 0.5B (base), Qwen 7B (base), Qwen2-7B-Instruct · method: Difference-in-means direction, Subspace-Compression Defense (FJLT / Bottleneck Autoencoder)
Representational Depth of Evaluation Awareness Shifts With Scale in Open-Weight LMs (2026)measured

Evaluation-awareness probe depth shifts late-to-early with model scale

Details

Manek projects a diff-of-means evaluation-awareness direction (from 203 contrastive prompt pairs) onto residual-stream activations at every layer across 11 open-weight models (Qwen2.5 0.5B-32B, Gemma2 2B/9B/27B, Llama-3.2 1B/3B) [manek-2026-representational-depth-of-evaluation-awareness-shifts-with-scale] The relative depth at which per-layer decoding AUROC peaks shifts from late layers in small models to the earliest layers in large ones (Qwen2.5 1.5B/3B peak at 0.96-0.97 vs 14B/32B at 0.021-0.031) [manek-2026-representational-depth-of-evaluation-awareness-shifts-with-scale] Gemma2 shows the same late-to-early shift (0.885 to 0.304) while Llama-3.2 stays mid-layer across both tested sizes [manek-2026-representational-depth-of-evaluation-awareness-shifts-with-scale] Peak AUROC ranges 0.586-0.873 and is non-monotonic with scale within a family [manek-2026-representational-depth-of-evaluation-awareness-shifts-with-scale]

models: Qwen2.5-0.5B-Instruct, Qwen2.5-1.5B, Qwen2.5-3B-Instruct, Qwen2.5-7B-Instruct, Qwen2.5-14B-Instruct, Qwen2.5-32B · method: Direction Extraction, Linear probing, Geometric analysis
Two Axes of LLM Abstention: Answer Correctness and Question Answerability (2026)measured

A probe detects unanswerability far better than confidence, not shrinking with scale

Details

Wagner fits logistic-regression probes on final-prompt-token hidden states of Gemma 2 2B-it, Qwen2.5-3B/7B/14B, and Llama-3.1-8B [wagner-2026-two-axes-of-llm-abstention-answer-correctness-and-question-answerability] Hidden-state answerability readout reaches AUROC 0.97-0.99 across all five models versus only 0.54-0.67 for output-confidence readouts, a gap that does not shrink with scale [wagner-2026-two-axes-of-llm-abstention-answer-correctness-and-question-answerability] On naturally-occurring false-premise questions all confidence signals stay near chance while the probe reaches 0.69-0.77 AUROC [wagner-2026-two-axes-of-llm-abstention-answer-correctness-and-question-answerability] Routing a premise-checking instruction through the probe roughly triples challenge precision; no causal steering is performed [wagner-2026-two-axes-of-llm-abstention-answer-correctness-and-question-answerability]

models: Qwen2.5-3B, Qwen2.5-7B, Qwen2.5-14B · method: Linear probing
Monitoring Emergent Reward Hacking During Generation via Internal Activations (2026)measured

A probe on SAE-PCA activations detects reward hacking token-by-token

Details

Wilhelm, Wittkopp & Kao train a per-layer sparse autoencoder on the last four layers of Qwen2.5-Instruct-7B, LLaMA-3.1-8B, and Falcon3-7B, then fit a logistic-regression probe on PCA-reduced SAE features for a per-token hack probability [wilhelm-wittkopp-kao-2026-monitoring-emergent-reward-hacking-during-generation-via-internal-activations] Against a GPT-4o judge (metric F1), the probe reaches F1=1.000 only on the benign control-adapter data, while reward-hacking detection F1 is 0.760-0.961 (e.g. Llama 0.961, Qwen 0.784) [wilhelm-wittkopp-kao-2026-monitoring-emergent-reward-hacking-during-generation-via-internal-activations] Reward-hacking versus benign activations are linearly separable in the SAE+PCA-reduced space; no causal steering is performed [wilhelm-wittkopp-kao-2026-monitoring-emergent-reward-hacking-during-generation-via-internal-activations]

models: Qwen2.5-7B-Instruct · method: Sparse Autoencoders (SAE), PCA, Linear probing
Decoding Emotion in the Deep: A Systematic Study of How LLMs Represent, Retain, and Express Emotion (2025)measured

Emotion clusters peak at mid-depth and tighten with scale

Details

Zhang & Zhong train two-layer MLP probes on frozen hidden states of Qwen3 (0.6B-8B) and LLaMA-3 (1B-8B), finding 7-class emotion clusters tight and well-separated for larger models but diffuse for smaller [zhang-zhong-2025-decoding-emotion-in-the-deep] Layer-wise probe accuracy rises from chance (0.143) to a peak of 0.78-0.80 at 50-75% relative depth rather than the final layer [zhang-zhong-2025-decoding-emotion-in-the-deep] An offset-aware probe shows emotion-specific signal decaying differentially across up to 400 generated tokens [zhang-zhong-2025-decoding-emotion-in-the-deep]

models: Qwen3-0.6B, Qwen3-1.7B, Qwen3-4B, Qwen3-8B · method: Mlp probing, PCA

Eternis Forecaster

What LLM Forecasters Know but Don't Say: Probing Internal Representations for Calibration and Faithfulness (2026)measured

A covariance-pooling probe beats verbalized confidence and catches stealth belief shifts

Details

Sarfati et al. train pooling probes on frozen residual-stream activations of Eternis-Forecaster-8B, GLM-4.5-Air, and Qwen3-8B to read out forecast correctness and confidence [sarfati-etal-2026-what-llm-forecasters-know-but-dont-say-probing-internal-representations-for-calibration-and-faithfulness] A layer-21 covariance probe halves calibration error (ECE 0.044 vs 0.093 for verbalized confidence at matched AUROC ~0.756) and holds out-of-distribution where verbalized AUROC collapses to 0.587 [sarfati-etal-2026-what-llm-forecasters-know-but-dont-say-probing-internal-representations-for-calibration-and-faithfulness] The probe's activation shift under evidence ablation/injection tracks behavioral change at Spearman rho=0.565 (vs 0.215 for reasoning-text changes), predicting the direction in 83.6% of cases including 107 stealth cases where the CoT shows no change [sarfati-etal-2026-what-llm-forecasters-know-but-dont-say-probing-internal-representations-for-calibration-and-faithfulness]

models: Eternis-Forecaster-8B (RLVR post-trained from Qwen3-8B) · method: Linear probing

GLM-4

What LLM Forecasters Know but Don't Say: Probing Internal Representations for Calibration and Faithfulness (2026)measured

A covariance-pooling probe beats verbalized confidence and catches stealth belief shifts

Details

Sarfati et al. train pooling probes on frozen residual-stream activations of Eternis-Forecaster-8B, GLM-4.5-Air, and Qwen3-8B to read out forecast correctness and confidence [sarfati-etal-2026-what-llm-forecasters-know-but-dont-say-probing-internal-representations-for-calibration-and-faithfulness] A layer-21 covariance probe halves calibration error (ECE 0.044 vs 0.093 for verbalized confidence at matched AUROC ~0.756) and holds out-of-distribution where verbalized AUROC collapses to 0.587 [sarfati-etal-2026-what-llm-forecasters-know-but-dont-say-probing-internal-representations-for-calibration-and-faithfulness] The probe's activation shift under evidence ablation/injection tracks behavioral change at Spearman rho=0.565 (vs 0.215 for reasoning-text changes), predicting the direction in 83.6% of cases including 107 stealth cases where the CoT shows no change [sarfati-etal-2026-what-llm-forecasters-know-but-dont-say-probing-internal-representations-for-calibration-and-faithfulness]

models: GLM-4.5-Air · method: Linear probing

DINOv3

Human-like Object Grouping in Self-Supervised Vision Transformers (2026)measured

Patch-cosine Gram structure groups objects, strongest in self-supervised ViTs

Details

Adeli et al. compute patch-token affinity maps and the full Gram matrix of pairwise patch cosine similarities from ViTs (DINOv3 ViT-B, DINOv2, DINO, MAE, supervised ViT-B, ConvNeXt-B) [adeli-etal-2026-human-like-object-grouping-in-self-supervised-vision-transformers] The Gram matrix shows block-like clustering of patches from the same object, quantified by AUC against ground-truth boundaries and a human-aligned grouping-accuracy benchmark [adeli-etal-2026-human-like-object-grouping-in-self-supervised-vision-transformers] Self-supervised training yields substantially more human-like grouping than supervised at matched architecture (DINOv3 ViT-B 91.9%, DINOv2 89.0% vs supervised ViT-B 70.6-72.2%) [adeli-etal-2026-human-like-object-grouping-in-self-supervised-vision-transformers] DINOv3-distilled ConvNeXt-B reaches 86.7% versus 60.0-67.4% for plain supervised ConvNeXt [adeli-etal-2026-human-like-object-grouping-in-self-supervised-vision-transformers]

models: DINOv3 ViT-B/16 · method: Geometric analysis

DINOv2

Human-like Object Grouping in Self-Supervised Vision Transformers (2026)measured

Patch-cosine Gram structure groups objects, strongest in self-supervised ViTs

Details

Adeli et al. compute patch-token affinity maps and the full Gram matrix of pairwise patch cosine similarities from ViTs (DINOv3 ViT-B, DINOv2, DINO, MAE, supervised ViT-B, ConvNeXt-B) [adeli-etal-2026-human-like-object-grouping-in-self-supervised-vision-transformers] The Gram matrix shows block-like clustering of patches from the same object, quantified by AUC against ground-truth boundaries and a human-aligned grouping-accuracy benchmark [adeli-etal-2026-human-like-object-grouping-in-self-supervised-vision-transformers] Self-supervised training yields substantially more human-like grouping than supervised at matched architecture (DINOv3 ViT-B 91.9%, DINOv2 89.0% vs supervised ViT-B 70.6-72.2%) [adeli-etal-2026-human-like-object-grouping-in-self-supervised-vision-transformers] DINOv3-distilled ConvNeXt-B reaches 86.7% versus 60.0-67.4% for plain supervised ConvNeXt [adeli-etal-2026-human-like-object-grouping-in-self-supervised-vision-transformers]

models: DINOv2 ViT-B/14 · method: Geometric analysis
From Local Cues to Global Percepts: Emergent Gestalt Organization in Self-Supervised Vision Models (2025)measured

A shared PCA basis's leading components separate figure from ground

Details

Li et al. project activations of several pretrained self-supervised vision models (ViT-MAE, CLIP ViT-H/14, DINOv2, ConvNeXt-MAE) onto the first 1-3 components of a fixed shared PCA basis [li-etal-2025-from-local-cues-to-global-percepts-emergent-gestalt-organization-in-self-supervised-vision-models] This low-dimensional axis cleanly separates figure from ground pixels, validated against ground-truth segmentation masks on 1,495 PASCAL VOC images [li-etal-2025-from-local-cues-to-global-percepts-emergent-gestalt-organization-in-self-supervised-vision-models] A Top-K activation-sparsity intervention raises a texture-oddity benchmark from 69.8 to 94.6 for MAE ViT and 62.1 to 88.1 for ConvNeXt-V1 [li-etal-2025-from-local-cues-to-global-percepts-emergent-gestalt-organization-in-self-supervised-vision-models]

models: DINOv2 ViT-B/14 · method: PCA

DINO

Human-like Object Grouping in Self-Supervised Vision Transformers (2026)measured

Patch-cosine Gram structure groups objects, strongest in self-supervised ViTs

Details

Adeli et al. compute patch-token affinity maps and the full Gram matrix of pairwise patch cosine similarities from ViTs (DINOv3 ViT-B, DINOv2, DINO, MAE, supervised ViT-B, ConvNeXt-B) [adeli-etal-2026-human-like-object-grouping-in-self-supervised-vision-transformers] The Gram matrix shows block-like clustering of patches from the same object, quantified by AUC against ground-truth boundaries and a human-aligned grouping-accuracy benchmark [adeli-etal-2026-human-like-object-grouping-in-self-supervised-vision-transformers] Self-supervised training yields substantially more human-like grouping than supervised at matched architecture (DINOv3 ViT-B 91.9%, DINOv2 89.0% vs supervised ViT-B 70.6-72.2%) [adeli-etal-2026-human-like-object-grouping-in-self-supervised-vision-transformers] DINOv3-distilled ConvNeXt-B reaches 86.7% versus 60.0-67.4% for plain supervised ConvNeXt [adeli-etal-2026-human-like-object-grouping-in-self-supervised-vision-transformers]

models: DINO ViT-B/16 · method: Geometric analysis

Vision Transformer (ViT)

Human-like Object Grouping in Self-Supervised Vision Transformers (2026)measured

Patch-cosine Gram structure groups objects, strongest in self-supervised ViTs

Details

Adeli et al. compute patch-token affinity maps and the full Gram matrix of pairwise patch cosine similarities from ViTs (DINOv3 ViT-B, DINOv2, DINO, MAE, supervised ViT-B, ConvNeXt-B) [adeli-etal-2026-human-like-object-grouping-in-self-supervised-vision-transformers] The Gram matrix shows block-like clustering of patches from the same object, quantified by AUC against ground-truth boundaries and a human-aligned grouping-accuracy benchmark [adeli-etal-2026-human-like-object-grouping-in-self-supervised-vision-transformers] Self-supervised training yields substantially more human-like grouping than supervised at matched architecture (DINOv3 ViT-B 91.9%, DINOv2 89.0% vs supervised ViT-B 70.6-72.2%) [adeli-etal-2026-human-like-object-grouping-in-self-supervised-vision-transformers] DINOv3-distilled ConvNeXt-B reaches 86.7% versus 60.0-67.4% for plain supervised ConvNeXt [adeli-etal-2026-human-like-object-grouping-in-self-supervised-vision-transformers]

models: MAE ViT-Base (Masked Autoencoder), ViT-Base · method: Geometric analysis
From Local Cues to Global Percepts: Emergent Gestalt Organization in Self-Supervised Vision Models (2025)measured

A shared PCA basis's leading components separate figure from ground

Details

Li et al. project activations of several pretrained self-supervised vision models (ViT-MAE, CLIP ViT-H/14, DINOv2, ConvNeXt-MAE) onto the first 1-3 components of a fixed shared PCA basis [li-etal-2025-from-local-cues-to-global-percepts-emergent-gestalt-organization-in-self-supervised-vision-models] This low-dimensional axis cleanly separates figure from ground pixels, validated against ground-truth segmentation masks on 1,495 PASCAL VOC images [li-etal-2025-from-local-cues-to-global-percepts-emergent-gestalt-organization-in-self-supervised-vision-models] A Top-K activation-sparsity intervention raises a texture-oddity benchmark from 69.8 to 94.6 for MAE ViT and 62.1 to 88.1 for ConvNeXt-V1 [li-etal-2025-from-local-cues-to-global-percepts-emergent-gestalt-organization-in-self-supervised-vision-models]

models: MAE ViT-Base (Masked Autoencoder) · method: PCA

ConvNeXt

Human-like Object Grouping in Self-Supervised Vision Transformers (2026)measured

Patch-cosine Gram structure groups objects, strongest in self-supervised ViTs

Details

Adeli et al. compute patch-token affinity maps and the full Gram matrix of pairwise patch cosine similarities from ViTs (DINOv3 ViT-B, DINOv2, DINO, MAE, supervised ViT-B, ConvNeXt-B) [adeli-etal-2026-human-like-object-grouping-in-self-supervised-vision-transformers] The Gram matrix shows block-like clustering of patches from the same object, quantified by AUC against ground-truth boundaries and a human-aligned grouping-accuracy benchmark [adeli-etal-2026-human-like-object-grouping-in-self-supervised-vision-transformers] Self-supervised training yields substantially more human-like grouping than supervised at matched architecture (DINOv3 ViT-B 91.9%, DINOv2 89.0% vs supervised ViT-B 70.6-72.2%) [adeli-etal-2026-human-like-object-grouping-in-self-supervised-vision-transformers] DINOv3-distilled ConvNeXt-B reaches 86.7% versus 60.0-67.4% for plain supervised ConvNeXt [adeli-etal-2026-human-like-object-grouping-in-self-supervised-vision-transformers]

models: ConvNeXt (image classifier, various sizes, ImageNet) · method: Geometric analysis
From Local Cues to Global Percepts: Emergent Gestalt Organization in Self-Supervised Vision Models (2025)measured

A shared PCA basis's leading components separate figure from ground

Details

Li et al. project activations of several pretrained self-supervised vision models (ViT-MAE, CLIP ViT-H/14, DINOv2, ConvNeXt-MAE) onto the first 1-3 components of a fixed shared PCA basis [li-etal-2025-from-local-cues-to-global-percepts-emergent-gestalt-organization-in-self-supervised-vision-models] This low-dimensional axis cleanly separates figure from ground pixels, validated against ground-truth segmentation masks on 1,495 PASCAL VOC images [li-etal-2025-from-local-cues-to-global-percepts-emergent-gestalt-organization-in-self-supervised-vision-models] A Top-K activation-sparsity intervention raises a texture-oddity benchmark from 69.8 to 94.6 for MAE ViT and 62.1 to 88.1 for ConvNeXt-V1 [li-etal-2025-from-local-cues-to-global-percepts-emergent-gestalt-organization-in-self-supervised-vision-models]

models: ConvNeXt (image classifier, various sizes, ImageNet) · method: PCA

Inception

Understanding Intermediate Layers Using Linear Classifier Probes (2016)measured

Linear-probe separability increases monotonically with CNN depth

Details

Alain & Bengio introduce the linear-classifier-probe method, fitting an independent linear probe at each layer of trained Inception-v3 and ResNet-50 [alain-bengio-2016-linear-classifier-probes] Probe accuracy, a linear-separability measurement, increases monotonically with network depth [alain-bengio-2016-linear-classifier-probes] This foundational work establishes linear probing as a standard representation-analysis technique and is purely observational [alain-bengio-2016-linear-classifier-probes] The authors caution probe accuracy is only a lower bound on the information present and can be confounded by probe capacity [alain-bengio-2016-linear-classifier-probes]

models: Inception V3 (ImageNet-trained) · method: Linear probing

ResNet

Understanding Intermediate Layers Using Linear Classifier Probes (2016)measured

Linear-probe separability increases monotonically with CNN depth

Details

Alain & Bengio introduce the linear-classifier-probe method, fitting an independent linear probe at each layer of trained Inception-v3 and ResNet-50 [alain-bengio-2016-linear-classifier-probes] Probe accuracy, a linear-separability measurement, increases monotonically with network depth [alain-bengio-2016-linear-classifier-probes] This foundational work establishes linear probing as a standard representation-analysis technique and is purely observational [alain-bengio-2016-linear-classifier-probes] The authors caution probe accuracy is only a lower bound on the information present and can be confounded by probe capacity [alain-bengio-2016-linear-classifier-probes]

models: ResNet-50 (supervised, ImageNet) · method: Linear probing
Learning Efficient Coding of Natural Images with Maximum Manifold Capacity Representations (2023)measured

MMCR trains competitive SSL via a distinct low-dim, large-radius geometry

Details

Yerxa et al. derive a differentiable manifold-capacity objective (negative nuclear norm of augmentation centroids) and train ResNet-50 on ImageNet, reaching 69.5-72.1% linear-eval accuracy matching SimCLR/MoCo-v2/BYOL/SwAV [yerxa-etal-2023-maximum-manifold-capacity-representations] MFTMA analysis shows MMCR achieves this via a distinct geometry: lower manifold dimensionality but larger radius, emerging in late representational stages [yerxa-etal-2023-maximum-manifold-capacity-representations] MMCR yields the highest participation ratio (279.2) and a spectral decay coefficient (1.04) closest to the target of 1 among six SSL methods, and is competitive on macaque V2/V4 neural predictivity [yerxa-etal-2023-maximum-manifold-capacity-representations]

models: ResNet-50 (Maximum Manifold Capacity Representations, self-supervised, ImageNet), ResNet-50 (MoCo v2, unsupervised contrastive pretraining, ImageNet) · method: Maximum Manifold Capacity Representations (MMCR) objective, Mean-Field Theoretic Manifold Analysis (MFTMA)
Separability and Geometry of Object Manifolds in Deep Neural Networks (2020)measured

Object manifolds untangle across CNN depth, causally driven by geometry

Details

Cohen et al. apply replica mean-field manifold-capacity theory to ImageNet object manifolds in AlexNet, VGG-16, and ResNet-50 [cohen-etal-2020-separability-geometry-object-manifolds] Classification capacity increases substantially along the trained hierarchy, up to nearly two orders of magnitude over shuffled-label controls for high-variability smooth manifolds [cohen-etal-2020-separability-geometry-object-manifolds] Manifold dimension and radius shrink and inter-manifold center correlations decrease with depth, none of which occur in untrained or label-shuffled controls, isolating training as the cause [cohen-etal-2020-separability-geometry-object-manifolds] Three manifold-perturbation interventions (size scaling, ball approximation, center randomization) confirm dimension reduction drives 55-90% of the capacity gain [cohen-etal-2020-separability-geometry-object-manifolds]

models: ResNet (image classifier, various depths) · method: Mean-Field Theoretic Manifold Analysis (MFTMA)

GPT-OSS

A Behavioural and Representational Evaluation of Goal-Directedness in Language Model Agents (2026)measured

Probes decode a grid cognitive map that reasoning then reorganizes

Details

Arghal et al. use linear and MLP probes to decode a grid-position and goal-location cognitive map from GPT-OSS-20B's layer-15 pre-reasoning activations in a 2D grid-world navigation task [arghal-etal-2026-a-behavioural-and-representational-evaluation-of-goal-directedness-in-language-model-agents] The agent's chosen action agrees with the decoded map at 82.5% average accuracy across grid sizes [arghal-etal-2026-a-behavioural-and-representational-evaluation-of-goal-directedness-in-language-model-agents] Localization accuracy degrades with grid size, and post-reasoning activations reorganize the cognitive-map signal toward immediate action selection rather than a stable spatial code [arghal-etal-2026-a-behavioural-and-representational-evaluation-of-goal-directedness-in-language-model-agents]

models: GPT-OSS-20B · method: Linear probing, Mlp probing
How Language Directions Align with Token Geometry in Multilingual LLMs (2026)measured

Language identity is linearly separable from layer 1; alignment tracks pretraining mix

Details

Kim & Lee probe all layers of six multilingual LLMs across five XNLI languages, training linear and MLP probes to classify language identity [kim-lee-2026-language-directions-token-geometry-multilingual-llms] Language separability jumps sharply in the first transformer block (+76.4 points) and stays almost fully linearly separable throughout depth (linear-probe accuracy 99.8%, only 0.58 points below the MLP) [kim-lee-2026-language-directions-token-geometry-multilingual-llms] A Token-Language Alignment metric tracks pretraining composition: Chinese-inclusive models reach 16.43% Chinese Match@Peak versus 3.90% for English-centric ones (4.21x) [kim-lee-2026-language-directions-token-geometry-multilingual-llms] Latin-script languages show uniformly low alignment regardless of model, confounded by shared script; no causal intervention is performed [kim-lee-2026-language-directions-token-geometry-multilingual-llms]

models: GPT-OSS-20B · method: Linear probing
What Really Controls Temporal Reasoning in LLMs: Tokenisation or Representation of Time? (2026)measured

Temporal-probe decodability predicts reasoning in high-resource languages

Details

Bhatia et al. introduce MultiTempBench and evaluate 20 LLMs, combining a tokenization-fragmentation metric with linear-regression probes decoding Year/Month/Day (probe R^2 as temporal linearity) [bhatia-etal-2026-what-really-controls-temporal-reasoning-in-llms] A crossed mixed-effects regression over 285,000 predictions finds temporal linearity is the strongest predictor of reasoning accuracy in high-resource languages (English r=0.77) [bhatia-etal-2026-what-really-controls-temporal-reasoning-in-llms] Tokenization fragmentation dominates in low-resource languages (Hausa r=-0.97); the finding is purely correlational [bhatia-etal-2026-what-really-controls-temporal-reasoning-in-llms]

models: GPT-OSS-20B · method: Linear probing

OLMo

Proxy Reward Internalization and Mechanistic Exploitation: A Learned Precursor to Reward Hacking and Its Generalization (2026)measured

Diff-in-means reward-hacking directions forecast and causally control misalignment

Details

Beigi, Jin & Huang extract three diff-in-means directions (context-sycophantic-agreement, premise-rejection, evaluator-rejection) during RL training of Qwen2.5-Coder-14B-Instruct and track their projection score across checkpoints [beigi-jin-huang-2026-proxy-reward-internalization-and-mechanistic-exploitation] Each direction is linearly decodable (AUROC 0.85-0.90) and a linear fit from probe score forecasts future out-of-domain misalignment (R^2=0.77), preceding the misalignment rise by roughly 45 training steps [beigi-jin-huang-2026-proxy-reward-internalization-and-mechanistic-exploitation] Joint ablation of the three directions lowers the hack rate by 26 points while preserving coding accuracy, and injection along them increases hacking [beigi-jin-huang-2026-proxy-reward-internalization-and-mechanistic-exploitation] The probe methodology generalizes to Qwen2.5-Coder 1.5B/7B/32B, Llama-3.1-8B-Instruct, and OLMo-7B [beigi-jin-huang-2026-proxy-reward-internalization-and-mechanistic-exploitation]

models: OLMo-7B · method: Difference-in-means direction
What Really Controls Temporal Reasoning in LLMs: Tokenisation or Representation of Time? (2026)measured

Temporal-probe decodability predicts reasoning in high-resource languages

Details

Bhatia et al. introduce MultiTempBench and evaluate 20 LLMs, combining a tokenization-fragmentation metric with linear-regression probes decoding Year/Month/Day (probe R^2 as temporal linearity) [bhatia-etal-2026-what-really-controls-temporal-reasoning-in-llms] A crossed mixed-effects regression over 285,000 predictions finds temporal linearity is the strongest predictor of reasoning accuracy in high-resource languages (English r=0.77) [bhatia-etal-2026-what-really-controls-temporal-reasoning-in-llms] Tokenization fragmentation dominates in low-resource languages (Hausa r=-0.97); the finding is purely correlational [bhatia-etal-2026-what-really-controls-temporal-reasoning-in-llms]

models: OLMo 3 7B · method: Linear probing

ESM-2

Viral Proteins Reveal Geometry of Protein Language Models (2026)measured

A protein-LM nativeness axis orders sequences; viral signal stays separable

Details

Bigot et al. run PCA on mean-pooled embeddings from five protein language models (ESM2, ESM3, ESMC, ProGen2, EvoDiff), finding a dominant nativeness axis (PC1, 54.3-73.1% variance) correlating with masked-reconstruction perplexity (rho=0.83-0.96) [bigot-etal-2026-viral-proteins-reveal-geometry-of-protein-language-models] The axis replicates within each of 10 biological groups via within-group PCA refits [bigot-etal-2026-viral-proteins-reveal-geometry-of-protein-language-models] Linear probes classify viral versus cellular proteins at AUC 0.97-1.00 even under homology-controlled splits, with 88.3-96.7% true-positive rate at 1% false-positive rate versus 26.9-39.2% for a perplexity-only classifier [bigot-etal-2026-viral-proteins-reveal-geometry-of-protein-language-models]

models: ESM-2 (650M), ESM-2 (15B) · method: PCA, Linear probing
Protein Contacts Are Already in the Attention: A Single-Forward-Pass Alternative to the Categorical Jacobian (2026)measured

ESM attention maps are linearly readable as contact maps in one pass

Details

Thorstenson shows raw attention matrices from ESM-2 (35M-3B) and ESMC-600M, combined via a parameter-free mean read-out, recover residue-residue contact maps [thorstenson-2026-protein-contacts-are-already-in-the-attention] Accuracy is competitive with the categorical-Jacobian method, which requires an expensive combinatorial sweep of masked-residue mutations per protein [thorstenson-2026-protein-contacts-are-already-in-the-attention] Because the read-out is parameter-free and needs only a single forward pass, contact information is already linearly present in the raw attention patterns [thorstenson-2026-protein-contacts-are-already-in-the-attention]

models: ESM-2 (35M), ESM-2 (150M), ESM-2 (650M), ESM-2 (3B) · method: Geometric analysis

ESM3

Viral Proteins Reveal Geometry of Protein Language Models (2026)measured

A protein-LM nativeness axis orders sequences; viral signal stays separable

Details

Bigot et al. run PCA on mean-pooled embeddings from five protein language models (ESM2, ESM3, ESMC, ProGen2, EvoDiff), finding a dominant nativeness axis (PC1, 54.3-73.1% variance) correlating with masked-reconstruction perplexity (rho=0.83-0.96) [bigot-etal-2026-viral-proteins-reveal-geometry-of-protein-language-models] The axis replicates within each of 10 biological groups via within-group PCA refits [bigot-etal-2026-viral-proteins-reveal-geometry-of-protein-language-models] Linear probes classify viral versus cellular proteins at AUC 0.97-1.00 even under homology-controlled splits, with 88.3-96.7% true-positive rate at 1% false-positive rate versus 26.9-39.2% for a perplexity-only classifier [bigot-etal-2026-viral-proteins-reveal-geometry-of-protein-language-models]

models: ESM3 (1.4B, OPEN), ESM3 (98B, LARGE) · method: PCA, Linear probing

ESM Cambrian (ESMC)

Viral Proteins Reveal Geometry of Protein Language Models (2026)measured

A protein-LM nativeness axis orders sequences; viral signal stays separable

Details

Bigot et al. run PCA on mean-pooled embeddings from five protein language models (ESM2, ESM3, ESMC, ProGen2, EvoDiff), finding a dominant nativeness axis (PC1, 54.3-73.1% variance) correlating with masked-reconstruction perplexity (rho=0.83-0.96) [bigot-etal-2026-viral-proteins-reveal-geometry-of-protein-language-models] The axis replicates within each of 10 biological groups via within-group PCA refits [bigot-etal-2026-viral-proteins-reveal-geometry-of-protein-language-models] Linear probes classify viral versus cellular proteins at AUC 0.97-1.00 even under homology-controlled splits, with 88.3-96.7% true-positive rate at 1% false-positive rate versus 26.9-39.2% for a perplexity-only classifier [bigot-etal-2026-viral-proteins-reveal-geometry-of-protein-language-models]

models: ESMC (600M), ESMC (6B) · method: PCA, Linear probing
Protein Contacts Are Already in the Attention: A Single-Forward-Pass Alternative to the Categorical Jacobian (2026)measured

ESM attention maps are linearly readable as contact maps in one pass

Details

Thorstenson shows raw attention matrices from ESM-2 (35M-3B) and ESMC-600M, combined via a parameter-free mean read-out, recover residue-residue contact maps [thorstenson-2026-protein-contacts-are-already-in-the-attention] Accuracy is competitive with the categorical-Jacobian method, which requires an expensive combinatorial sweep of masked-residue mutations per protein [thorstenson-2026-protein-contacts-are-already-in-the-attention] Because the read-out is parameter-free and needs only a single forward pass, contact information is already linearly present in the raw attention patterns [thorstenson-2026-protein-contacts-are-already-in-the-attention]

models: ESMC (600M) · method: Geometric analysis

ProGen2

Viral Proteins Reveal Geometry of Protein Language Models (2026)measured

A protein-LM nativeness axis orders sequences; viral signal stays separable

Details

Bigot et al. run PCA on mean-pooled embeddings from five protein language models (ESM2, ESM3, ESMC, ProGen2, EvoDiff), finding a dominant nativeness axis (PC1, 54.3-73.1% variance) correlating with masked-reconstruction perplexity (rho=0.83-0.96) [bigot-etal-2026-viral-proteins-reveal-geometry-of-protein-language-models] The axis replicates within each of 10 biological groups via within-group PCA refits [bigot-etal-2026-viral-proteins-reveal-geometry-of-protein-language-models] Linear probes classify viral versus cellular proteins at AUC 0.97-1.00 even under homology-controlled splits, with 88.3-96.7% true-positive rate at 1% false-positive rate versus 26.9-39.2% for a perplexity-only classifier [bigot-etal-2026-viral-proteins-reveal-geometry-of-protein-language-models]

models: ProGen2-base · method: PCA, Linear probing

EvoDiff

Viral Proteins Reveal Geometry of Protein Language Models (2026)measured

A protein-LM nativeness axis orders sequences; viral signal stays separable

Details

Bigot et al. run PCA on mean-pooled embeddings from five protein language models (ESM2, ESM3, ESMC, ProGen2, EvoDiff), finding a dominant nativeness axis (PC1, 54.3-73.1% variance) correlating with masked-reconstruction perplexity (rho=0.83-0.96) [bigot-etal-2026-viral-proteins-reveal-geometry-of-protein-language-models] The axis replicates within each of 10 biological groups via within-group PCA refits [bigot-etal-2026-viral-proteins-reveal-geometry-of-protein-language-models] Linear probes classify viral versus cellular proteins at AUC 0.97-1.00 even under homology-controlled splits, with 88.3-96.7% true-positive rate at 1% false-positive rate versus 26.9-39.2% for a perplexity-only classifier [bigot-etal-2026-viral-proteins-reveal-geometry-of-protein-language-models]

models: EvoDiff OA-DM · method: PCA, Linear probing

InternVL

Steering and Rectifying Latent Representation Manifolds in Frozen Multi-modal LLMs for Video Anomaly Detection (2026)measured

A per-head Fisher-ratio score selects steering that nearly matches fine-tuning

Details

Cai et al. define a per-attention-head Representational Separability Analysis score (inter- over intra-cluster scatter) quantifying normal/anomalous manifold entanglement in a frozen InternVL3-8B [cai-etal-2026-steervad-manifold-rectification] The score selects which heads receive a targeted anisotropic-scaling causal intervention that stretches or compresses the manifold along discriminative axes [cai-etal-2026-steervad-manifold-rectification] SteerVAD reaches 87.15% AUC on UCF-Crime, nearly matching a fully fine-tuned baseline (Holmes-VAD 89.51%) with about 1% of its training data [cai-etal-2026-steervad-manifold-rectification] Ablating the anisotropic steering degrades AUC, and a linear-classifier-on-raw-features baseline reaches only 81.33% [cai-etal-2026-steervad-manifold-rectification]

models: InternVL3-8B · method: Representational Separability Analysis (inter-to-intra scatter ratio), Causal interventions (steering)
Encoded but Not Routed: Explaining the Table-Chart Gap in Scientific Claim Verification (2026)measured

VLMs encode chart signal linearly but do not route it to decisions

Details

Kumar et al. train layer-wise logistic-regression probes on hidden states of Qwen2.5-VL-7B/32B-Instruct and InternVL3-8B for chart-claim verification [kumar-etal-2026-encoded-but-not-routed-explaining-the-table-chart-gap-in-scientific-claim-verification] On InternVL3-8B the mean-pool probe reaches 79.0% accuracy while the model's own decision reaches only 56.2%; on Qwen2.5-VL-32B probe AUROC is 88.9% versus 71.9% at the last token (McNemar p<0.01) [kumar-etal-2026-encoded-but-not-routed-explaining-the-table-chart-gap-in-scientific-claim-verification] An attention analysis finds the Qwen models attend to image tokens at only 4-11% of a proportional baseline at the final layer, while InternVL3 attends at about 93% [kumar-etal-2026-encoded-but-not-routed-explaining-the-table-chart-gap-in-scientific-claim-verification] The authors note the analysis is diagnostic, not causal [kumar-etal-2026-encoded-but-not-routed-explaining-the-table-chart-gap-in-scientific-claim-verification]

models: InternVL3-8B · method: Linear probing
Probing Visual Concepts in Lightweight Vision-Language Models for Automated Driving (2026)measured

Lightweight VLMs linearly encode object presence but never orientation

Details

Theodoridis et al. use CARLA counterfactual pairs and layer-wise linear probes across four lightweight VLMs (Ovis2.5-2B, InternVL3.5-2B, VST-3B SFT and RL) [theodoridis-etal-2026-probing-visual-concepts-in-lightweight-vision-language-models-for-automated-driving] Object/agent presence is linearly encoded from mid-vision-encoder through final LLM layers, while orientation is never linearly encoded anywhere, and increasing object distance rapidly degrades separability [theodoridis-etal-2026-probing-visual-concepts-in-lightweight-vision-language-models-for-automated-driving] Two failure modes are dissociated: perceptual (concept not encoded) versus cognitive (concept present but the model still answers wrong) [theodoridis-etal-2026-probing-visual-concepts-in-lightweight-vision-language-models-for-automated-driving] Activation steering along the probe directions causally confirms they drive behavior rather than being correlational [theodoridis-etal-2026-probing-visual-concepts-in-lightweight-vision-language-models-for-automated-driving]

models: InternVL3.5-2B · method: Supervised direction search (gradient-based optimization), Causal interventions (steering)

Mistral

Do LLMs Really Know What They Don't Know? Internal States Mainly Reflect Knowledge Recall Rather Than Truthfulness (2026)measured

Probes separate knowledge familiarity from truthfulness better than confidence

Details

Cheang et al. fit logistic-regression probes on subject-token, attention, and last-layer hidden states of LLaMA-3-8B (main) and Mistral-7B-v0.3 (replication) [cheang-etal-2026-do-llms-really-know-what-they-dont-know-internal-states-mainly-reflect-knowledge-recall-rather-than-truthfulness] A last-token probe reaches AUROC 0.69 distinguishing attributable from unattributable hallucination but 0.93 distinguishing unfamiliar from familiar entities; subject and attention probes show the same gap [cheang-etal-2026-do-llms-really-know-what-they-dont-know-internal-states-mainly-reflect-knowledge-recall-rather-than-truthfulness] Cluster-separability metrics (Silhouette, Davies-Bouldin on t-SNE) show knowledge-recall and truthfulness occupy measurably different regions, replicated on Mistral-7B-v0.3 [cheang-etal-2026-do-llms-really-know-what-they-dont-know-internal-states-mainly-reflect-knowledge-recall-rather-than-truthfulness]

models: Mistral-7B-v0.3 · method: Linear probing
Hallucination Basins: A Dynamic Framework for Understanding and Controlling LLM Hallucinations (2026)measured

A diff-of-means hallucination direction causally raises hallucination dose-dependently

Details

Cherukuri & Varshney formalize hallucinations as basin attractors across Llama-3.2-1B/3B, Gemma-2-2B, Qwen2.5-1.5B, Llama-3.1-8B, and Mistral-7B-v0.3 [cherukuri-varshney-2026-hallucination-basins-a-dynamic-framework-for-understanding-and-controlling-llm-hallucinations] Basin separation is strongly task-dependent: factoid tasks show sharp centroid separation (variance ratio up to 4.55, AUROC up to 1.000 on MuSiQue) while summarization and TruthfulQA hover near chance [cherukuri-varshney-2026-hallucination-basins-a-dynamic-framework-for-understanding-and-controlling-llm-hallucinations] Interpolating factual hidden states toward the hallucination centroid via a diff-of-means steering vector produces a monotonic dose-response increase in hallucination probability, exceeding random and orthogonal controls [cherukuri-varshney-2026-hallucination-basins-a-dynamic-framework-for-understanding-and-controlling-llm-hallucinations]

models: Mistral-7B-v0.3 · method: Difference-in-means direction, Activation Steering (Addition)
Large Language Models Encode Semantics and Alignment in Linearly Separable Representations (2025)measured

Topic subspaces sharpen with depth; a centroid direction induces CoT

Details

Saglam et al. study 11 autoregressive models on six arXiv topics, measuring per-layer hard-margin SVM separability and PCA intrinsic dimensionality [saglam-etal-2025-linear-subspaces] Under 10% of principal components explain nearly all variance, so topics occupy a near-affine low-dimensional subspace; separability rises toward final layers, reaching 100% SVM accuracy on all topic pairs in the largest models [saglam-etal-2025-linear-subspaces] A chain-of-thought framing distinction becomes linearly separable even more sharply than topic identity [saglam-etal-2025-linear-subspaces] Adding the CoT-vs-non-CoT centroid-difference direction to the hidden state reliably induces CoT-style responses (flagged preliminary), and a lightweight MLP guardrail built on the finding halves harmful responses [saglam-etal-2025-linear-subspaces]

models: Mistral Small 3 (2501), Mistral-7B · method: Linear probing, PCA, Activation Steering (Addition)
Language Models Represent Beliefs of Self and Others (2024)measured

A mid-layer head group decodes another agent's beliefs and steers theory-of-mind

Details

Zhu et al. prompt Mistral-7B-Instruct (and DeepSeek-LLM-7B-Chat) with third-person belief narratives and fit logistic-regression probes on attention-head activations [zhu-etal-2024-language-models-represent-beliefs-of-self-and-others] A group of middle-layer attention heads decodes another agent's belief status at over 80% accuracy, distinct from the model's own self-belief representation [zhu-etal-2024-language-models-represent-beliefs-of-self-and-others] Steering these representations produces large changes in downstream Theory-of-Mind performance, generalizing across social-reasoning tasks [zhu-etal-2024-language-models-represent-beliefs-of-self-and-others]

models: Mistral-7B-Instruct · method: Linear probing, Activation Steering (Addition)

Gemma

Hallucination Basins: A Dynamic Framework for Understanding and Controlling LLM Hallucinations (2026)measured

A diff-of-means hallucination direction causally raises hallucination dose-dependently

Details

Cherukuri & Varshney formalize hallucinations as basin attractors across Llama-3.2-1B/3B, Gemma-2-2B, Qwen2.5-1.5B, Llama-3.1-8B, and Mistral-7B-v0.3 [cherukuri-varshney-2026-hallucination-basins-a-dynamic-framework-for-understanding-and-controlling-llm-hallucinations] Basin separation is strongly task-dependent: factoid tasks show sharp centroid separation (variance ratio up to 4.55, AUROC up to 1.000 on MuSiQue) while summarization and TruthfulQA hover near chance [cherukuri-varshney-2026-hallucination-basins-a-dynamic-framework-for-understanding-and-controlling-llm-hallucinations] Interpolating factual hidden states toward the hallucination centroid via a diff-of-means steering vector produces a monotonic dose-response increase in hallucination probability, exceeding random and orthogonal controls [cherukuri-varshney-2026-hallucination-basins-a-dynamic-framework-for-understanding-and-controlling-llm-hallucinations]

models: Gemma-2-2B · method: Difference-in-means direction, Activation Steering (Addition)
Finding Lexical Identity and Inflectional Morphology in Modern Language Models (2025)measured

Morphology stays linearly separable while lexical identity turns nonlinear with depth

Details

Li & Subramani train ridge-regression linear probes and nonlinear MLP probes across 25 pretrained models to predict word lemma and inflectional features [li-subramani-2025-finding-lexical-identity-and-inflectional-morphology-in-modern-language-models] Lemma is linearly separable in early layers (0.8-1.0) but linear accuracy declines toward final layers while MLP accuracy declines much less, so lexical identity becomes increasingly nonlinearly encoded [li-subramani-2025-finding-lexical-identity-and-inflectional-morphology-in-modern-language-models] Inflectional morphology stays linearly separable at 0.9-1.0 uniformly across every layer, with MLP probes adding no benefit [li-subramani-2025-finding-lexical-identity-and-inflectional-morphology-in-modern-language-models] Control-task selectivity confirms inflection is a generalizable linear abstraction (selectivity 0.4-0.6) while lemma decodability reflects near-zero-selectivity memorization [li-subramani-2025-finding-lexical-identity-and-inflectional-morphology-in-modern-language-models]

models: Gemma-2-2B · method: Linear probing, Mlp probing
What Really Controls Temporal Reasoning in LLMs: Tokenisation or Representation of Time? (2026)measured

Temporal-probe decodability predicts reasoning in high-resource languages

Details

Bhatia et al. introduce MultiTempBench and evaluate 20 LLMs, combining a tokenization-fragmentation metric with linear-regression probes decoding Year/Month/Day (probe R^2 as temporal linearity) [bhatia-etal-2026-what-really-controls-temporal-reasoning-in-llms] A crossed mixed-effects regression over 285,000 predictions finds temporal linearity is the strongest predictor of reasoning accuracy in high-resource languages (English r=0.77) [bhatia-etal-2026-what-really-controls-temporal-reasoning-in-llms] Tokenization fragmentation dominates in low-resource languages (Hausa r=-0.97); the finding is purely correlational [bhatia-etal-2026-what-really-controls-temporal-reasoning-in-llms]

models: Gemma 3 4B Instruct · method: Linear probing
Catching Rationalization in the Act: Detecting Motivated Reasoning Before and After CoT via Activation Probing (2026)measured

Probes detect motivated reasoning before and after chain-of-thought

Details

Mirtaheri & Belkin use a paired hinted/unhinted framework to label motivated reasoning across Qwen3-8B, Llama-3.1-8B-Instruct, and Gemma-3-4B, probing activations with recursive feature machines [mirtaheri-belkin-2026-catching-rationalization-in-the-act-detecting-motivated-reasoning-before-and-after-cot-via-activation-probing] Pre-generation probes match a GPT-5-nano full-trace CoT monitor and post-generation probes outperform it [mirtaheri-belkin-2026-catching-rationalization-in-the-act-detecting-motivated-reasoning-before-and-after-cot-via-activation-probing] A hint-recovery probe shows a U-shaped accuracy curve across CoT tokens, so the model internally re-engages the hinted answer even when its CoT never mentions it [mirtaheri-belkin-2026-catching-rationalization-in-the-act-detecting-motivated-reasoning-before-and-after-cot-via-activation-probing]

models: Gemma 3 4B Instruct · method: Recursive Feature Machine (RFM)
Unifying Attention Heads and Task Vectors via Hidden State Geometry in In-Context Learning (2025)measured

ICL separability rises first, then alignment with the unembedding

Details

Yang et al. bound classification accuracy by hidden-state linear separability, requiring both output alignment and directional alignment with the label unembedding-difference vector [yang-etal-2025] Across 7 models on 6 datasets, ICL proceeds in two stages: separability rises rapidly in early layers, then middle-to-late layers spike four alignment measures together [yang-etal-2025] The ICL-versus-zero-shot separability gap is small despite an ~80-point accuracy gap, so ICL's gains come almost entirely from improved alignment, not separability [yang-etal-2025] Ablating Previous-Token Heads collapses separability while sparing alignment, and ablating Induction Heads collapses alignment and accuracy (to 2.5%), giving a geometric account of function vectors [yang-etal-2025]

models: Gemma-2B, Gemma-7B · method: Linear probing, PCA, Causal interventions (steering), Function vector extraction (causal-mediation head selection + sum), Separability/alignment decomposition
The Obfuscation Atlas: Mapping Where Honesty Emerges in RLVR with Deception Probes (2026)measured

Layerwise honesty-decodability probes drift sharply off-domain under real RLVR training

Details

Taufeeque, Heimersheim, Gleave & Cundy train per-layer logistic-regression probes mapping where honesty/deception is linearly decodable in real Llama-3-8B-Instruct, Gemma-3-12B/27B-IT, and Llama-3-70B-Instruct, then run real LoRA+GRPO RLVR training on an MBPP-Honeypot coding environment where reward hacking naturally emerges [taufeeque-etal-2026-obfuscation-atlas-honesty-emerges-in-rlvr] Pre-RL probe AUC is near-ceiling (0.991-1.000); after RLVR training with no detector penalty, AUC on off-domain deception probes collapses (e.g. 0.991 to 0.376, 0.996 to 0.655) while on-domain MBPP-probe AUC stays at 1.000, indicating representation drift rather than deliberate evasion [taufeeque-etal-2026-obfuscation-atlas-honesty-emerges-in-rlvr] Adding an explicit detector penalty during training instead produces genuinely evasive deceptive behavior in 96.4 percent of seeds (27 of 28) [taufeeque-etal-2026-obfuscation-atlas-honesty-emerges-in-rlvr]

models: Gemma 3 27B Instruct · method:
The Blessing and Curse of Dimensionality in Safety Alignment (2025)measured

Linear safety separability emerges above a hidden-dimension threshold

Details

Teo, Abdullaev & Nguyen show via PCA across Qwen0.5B, GPT2-XL, Qwen7B, and Llama2-7B that emotion clusters become distinctly separable only above ~2,000-3,000 hidden dimensions [teo-etal-2025-blessing-and-curse-of-dimensionality-in-safety-alignment] This emergent linear separability is exactly what white-box jailbreaks exploit, collapsing Llama2-7B-Chat's refusal/safety from 0.97/0.99 to 0.03/0.14 [teo-etal-2025-blessing-and-curse-of-dimensionality-in-safety-alignment] Two dimension-reduction defenses (FJLT random projection and a learned bottleneck autoencoder) restore refusal/safety to 0.94-0.96, with the bottleneck retaining about 70% of benign helpfulness [teo-etal-2025-blessing-and-curse-of-dimensionality-in-safety-alignment] A Rademacher-complexity bound explains why shrinking the ambient dimension raises the sample cost of finding a steering direction [teo-etal-2025-blessing-and-curse-of-dimensionality-in-safety-alignment]

models: Gemma 1.1 7B Instruct · method: Difference-in-means direction, Subspace-Compression Defense (FJLT / Bottleneck Autoencoder)
Representational Depth of Evaluation Awareness Shifts With Scale in Open-Weight LMs (2026)measured

Evaluation-awareness probe depth shifts late-to-early with model scale

Details

Manek projects a diff-of-means evaluation-awareness direction (from 203 contrastive prompt pairs) onto residual-stream activations at every layer across 11 open-weight models (Qwen2.5 0.5B-32B, Gemma2 2B/9B/27B, Llama-3.2 1B/3B) [manek-2026-representational-depth-of-evaluation-awareness-shifts-with-scale] The relative depth at which per-layer decoding AUROC peaks shifts from late layers in small models to the earliest layers in large ones (Qwen2.5 1.5B/3B peak at 0.96-0.97 vs 14B/32B at 0.021-0.031) [manek-2026-representational-depth-of-evaluation-awareness-shifts-with-scale] Gemma2 shows the same late-to-early shift (0.885 to 0.304) while Llama-3.2 stays mid-layer across both tested sizes [manek-2026-representational-depth-of-evaluation-awareness-shifts-with-scale] Peak AUROC ranges 0.586-0.873 and is non-monotonic with scale within a family [manek-2026-representational-depth-of-evaluation-awareness-shifts-with-scale]

models: Gemma-2-2B, Gemma-2-9B, Gemma-2-27B · method: Direction Extraction, Linear probing, Geometric analysis
Large Language Models Encode Semantics and Alignment in Linearly Separable Representations (2025)measured

Topic subspaces sharpen with depth; a centroid direction induces CoT

Details

Saglam et al. study 11 autoregressive models on six arXiv topics, measuring per-layer hard-margin SVM separability and PCA intrinsic dimensionality [saglam-etal-2025-linear-subspaces] Under 10% of principal components explain nearly all variance, so topics occupy a near-affine low-dimensional subspace; separability rises toward final layers, reaching 100% SVM accuracy on all topic pairs in the largest models [saglam-etal-2025-linear-subspaces] A chain-of-thought framing distinction becomes linearly separable even more sharply than topic identity [saglam-etal-2025-linear-subspaces] Adding the CoT-vs-non-CoT centroid-difference direction to the hidden state reliably induces CoT-style responses (flagged preliminary), and a lightweight MLP guardrail built on the finding halves harmful responses [saglam-etal-2025-linear-subspaces]

models: Gemma-2-9B, Gemma-2-2B · method: Linear probing, PCA, Activation Steering (Addition)
Multilingual Language Models Encode Script Over Linguistic Structure (2026)measured

Language units are script-conditioned; typology decodes with depth

Details

Verma et al. use LAPE on MLP neurons and SAE-LAPE on latent features in Llama-3.2-1B and Gemma-2-2B to ask whether language units encode identity or surface script [verma-etal-2026-multilingual-lms-encode-script-over-linguistic-structure] Romanizing a non-Latin language produces a unit set nearly disjoint from both the native-script and English sets (Jaccard <0.3), a script-conditioned third subspace, while word-order shuffling leaves most raw-neuron units intact [verma-etal-2026-multilingual-lms-encode-script-over-linguistic-structure] Linear probing against lang2vec shows script-invariant units carry the strongest typological signal, and typological accessibility is depth-dependent (genealogy early, phonology deepest) [verma-etal-2026-multilingual-lms-encode-script-over-linguistic-structure] Causal ablation shows perplexity is most disrupted when script-invariant or order-invariant units are ablated, so functional necessity tracks surface-invariance rather than typological alignment [verma-etal-2026-multilingual-lms-encode-script-over-linguistic-structure]

models: Gemma-2-2B · method: Language Activation Probability Entropy (LAPE), Sparse Autoencoders (SAE), Causal interventions (steering)
Two Axes of LLM Abstention: Answer Correctness and Question Answerability (2026)measured

A probe detects unanswerability far better than confidence, not shrinking with scale

Details

Wagner fits logistic-regression probes on final-prompt-token hidden states of Gemma 2 2B-it, Qwen2.5-3B/7B/14B, and Llama-3.1-8B [wagner-2026-two-axes-of-llm-abstention-answer-correctness-and-question-answerability] Hidden-state answerability readout reaches AUROC 0.97-0.99 across all five models versus only 0.54-0.67 for output-confidence readouts, a gap that does not shrink with scale [wagner-2026-two-axes-of-llm-abstention-answer-correctness-and-question-answerability] On naturally-occurring false-premise questions all confidence signals stay near chance while the probe reaches 0.69-0.77 AUROC [wagner-2026-two-axes-of-llm-abstention-answer-correctness-and-question-answerability] Routing a premise-checking instruction through the probe roughly triples challenge precision; no causal steering is performed [wagner-2026-two-axes-of-llm-abstention-answer-correctness-and-question-answerability]

models: Gemma-2-2B-it · method: Linear probing

CLIP (Contrastive Language-Image Pretraining)

CLIP Behaves Like a Bag-of-Words Model Cross-modally but Not Uni-modally (2026)measured

CLIP binding is linearly decodable within a modality but collapses across

Details

Koishigarina et al. linearly probe CLIP (ViT-B/32, ViT-B/16, ViT-L/14) for attribute-object binding [koishigarina-etal-2026-clip-behaves-like-a-bag-of-words-model-cross-modally-but-not-uni-modally] Within a single modality (text-only or image-only) linear probes reach near-ceiling accuracy, so binding is linearly present uni-modally [koishigarina-etal-2026-clip-behaves-like-a-bag-of-words-model-cross-modally-but-not-uni-modally] Cross-modally, baseline CLIP binding accuracy collapses to near chance (0.51-0.52) [koishigarina-etal-2026-clip-behaves-like-a-bag-of-words-model-cross-modally-but-not-uni-modally] A lightweight learned linear map recovers cross-modal binding to 0.83-0.98 across the three variants, so the bottleneck is a cross-modal alignment gap rather than missing linear structure [koishigarina-etal-2026-clip-behaves-like-a-bag-of-words-model-cross-modally-but-not-uni-modally]

models: CLIP ViT-B/32, CLIP ViT-B/16, CLIP ViT-L/14 · method: Linear probing
From Local Cues to Global Percepts: Emergent Gestalt Organization in Self-Supervised Vision Models (2025)measured

A shared PCA basis's leading components separate figure from ground

Details

Li et al. project activations of several pretrained self-supervised vision models (ViT-MAE, CLIP ViT-H/14, DINOv2, ConvNeXt-MAE) onto the first 1-3 components of a fixed shared PCA basis [li-etal-2025-from-local-cues-to-global-percepts-emergent-gestalt-organization-in-self-supervised-vision-models] This low-dimensional axis cleanly separates figure from ground pixels, validated against ground-truth segmentation masks on 1,495 PASCAL VOC images [li-etal-2025-from-local-cues-to-global-percepts-emergent-gestalt-organization-in-self-supervised-vision-models] A Top-K activation-sparsity intervention raises a texture-oddity benchmark from 69.8 to 94.6 for MAE ViT and 62.1 to 88.1 for ConvNeXt-V1 [li-etal-2025-from-local-cues-to-global-percepts-emergent-gestalt-organization-in-self-supervised-vision-models]

models: CLIP ViT-H/14 · method: PCA

SimCLR

Linear Classification of Neural Manifolds with Correlated Variability (2023)measured

A correlated manifold-capacity formula beats the low-rank approximation

Details

Wakhloo, Sussman & Chung derive a closed-form manifold classification-capacity formula for object manifolds with arbitrary axis and centroid correlation, generalizing base manifold-capacity theory and its low-rank correction [wakhloo-etal-2023-correlated-neural-manifolds] Axis correlations act exactly like shrinking a manifold's radius while centroid correlations act like pulling centers together, a duality between correlation and geometry [wakhloo-etal-2023-correlated-neural-manifolds] On a SimCLR-trained ResNet-50, the new estimator tracks ground-truth simulated capacity more closely than the low-rank correction, which systematically overestimates capacity in later layers [wakhloo-etal-2023-correlated-neural-manifolds]

models: ResNet-50 (SimCLR contrastive pretraining, ImageNet) · method: Mean-Field Theoretic Manifold Analysis (MFTMA)
Learning Efficient Coding of Natural Images with Maximum Manifold Capacity Representations (2023)measured

MMCR trains competitive SSL via a distinct low-dim, large-radius geometry

Details

Yerxa et al. derive a differentiable manifold-capacity objective (negative nuclear norm of augmentation centroids) and train ResNet-50 on ImageNet, reaching 69.5-72.1% linear-eval accuracy matching SimCLR/MoCo-v2/BYOL/SwAV [yerxa-etal-2023-maximum-manifold-capacity-representations] MFTMA analysis shows MMCR achieves this via a distinct geometry: lower manifold dimensionality but larger radius, emerging in late representational stages [yerxa-etal-2023-maximum-manifold-capacity-representations] MMCR yields the highest participation ratio (279.2) and a spectral decay coefficient (1.04) closest to the target of 1 among six SSL methods, and is competitive on macaque V2/V4 neural predictivity [yerxa-etal-2023-maximum-manifold-capacity-representations]

models: ResNet-50 (SimCLR contrastive pretraining, ImageNet) · method: Maximum Manifold Capacity Representations (MMCR) objective, Mean-Field Theoretic Manifold Analysis (MFTMA)

Nucleotide Transformer

Frozen but Not Always Accessible: A Representation Analysis of Genomic Language Models (2026)measured

Frozen real genomic language models show task-dependent linear accessibility of biological signal, quantified by silhouette, kNN and Fisher-separation geometry

Details

Datta, Shatabda & Rahman (2026) probe five real frozen genomic language models (DNABERT-2, Nucleotide Transformer, HyenaDNA, GENERATOR-v2, Omni-DNA) with logistic-regression (linear), RBF-SVM and MLP (nonlinear) probes plus quantitative embedding-geometry metrics (silhouette score, 5-nearest-neighbor consistency, linear separability, Fisher separation ratio) on mean-pooled frozen embeddings [datta-shatabda-rahman-2026-frozen-but-not-always-accessible-genomic-language-models] Promoter-recognition signal is highly linearly accessible from frozen embeddings (frozen-probe recovery of fine-tuned performance: 98%/95%/100%/98% for HyenaDNA/NT/GENERATOR-v2/Omni-DNA), while splice-site signal is markedly less accessible (55-88% recovery), with Nucleotide Transformer's own embedding geometry showing the same split quantitatively: Promoter-All silhouette=0.044/kNN=0.883/linear-separability=0.917 versus Splice-All silhouette=-0.011/kNN=0.465/linear-separability=0.597 [datta-shatabda-rahman-2026-frozen-but-not-always-accessible-genomic-language-models] Layer-wise probing of Nucleotide Transformer on the Splice-All task shows intermediate layers (layer 20: ACC=0.639, F1=0.527, MCC=0.449) outperform the final layer (layer 24: ACC~0.60, F1~0.52); causal validation is via input-level in-silico mutagenesis (per-nucleotide substitution sensitivity), not internal activation patching [datta-shatabda-rahman-2026-frozen-but-not-always-accessible-genomic-language-models]

models: Nucleotide Transformer (500M) · method: Linear probing

Wan

The Invisible Hand of Physics: When Video Diffusion Models Know More Than They Show (2026)measured

Inverting video diffusion transformers makes physical plausibility linearly decodable

Details

Esmati et al. approximately invert the deterministic diffusion sampling of WAN-1.3B, CogVideoX-2B, and LTX-2B to recover intermediate states and attention maps [esmati-etal-2026-the-invisible-hand-of-physics] Physical plausibility is linearly decodable from the recovered states at ~81.27% average accuracy across IntPhys and InfLevel [esmati-etal-2026-the-invisible-hand-of-physics] This exceeds V-JEPA2 ViT-L and VideoMAE-Large baselines probed on their own forward-pass activations, so diffusion models implicitly encode physical structure recoverable only via inversion [esmati-etal-2026-the-invisible-hand-of-physics]

models: Wan2.1-1.3B · method: Diffusion-inversion probing, Linear probing

CogVideoX

The Invisible Hand of Physics: When Video Diffusion Models Know More Than They Show (2026)measured

Inverting video diffusion transformers makes physical plausibility linearly decodable

Details

Esmati et al. approximately invert the deterministic diffusion sampling of WAN-1.3B, CogVideoX-2B, and LTX-2B to recover intermediate states and attention maps [esmati-etal-2026-the-invisible-hand-of-physics] Physical plausibility is linearly decodable from the recovered states at ~81.27% average accuracy across IntPhys and InfLevel [esmati-etal-2026-the-invisible-hand-of-physics] This exceeds V-JEPA2 ViT-L and VideoMAE-Large baselines probed on their own forward-pass activations, so diffusion models implicitly encode physical structure recoverable only via inversion [esmati-etal-2026-the-invisible-hand-of-physics]

models: CogVideoX-2B, LTX-Video (diffusion-based video generator) · method: Diffusion-inversion probing, Linear probing
Do Video Foundation Models Understand Intuitive Physics? A Layerwise Probing Analysis (2026)measured

Video foundation models vary in physics decodability, V-JEPA strongest

Details

Punzo et al. fit frozen-feature linear probes layer-by-layer on V-JEPA, VideoMAE, and LTX-Video, evaluated on the IntPhys2 and Minimal Video Pairs intuitive-physics benchmarks [punzo-etal-2026-do-video-foundation-models-understand-intuitive-physics] V-JEPA (ViT-H) achieves the strongest results, especially with temporal-dynamics probes, VideoMAE is competitive, and LTX-Video recovers weaker but above-chance signal [punzo-etal-2026-do-video-foundation-models-understand-intuitive-physics] Physics-relevant information is weakest in early layers and peaks at intermediate-to-late depth [punzo-etal-2026-do-video-foundation-models-understand-intuitive-physics] Frame-order shuffling substantially degrades performance, confirming the signal reflects genuine temporal/physical structure rather than static appearance [punzo-etal-2026-do-video-foundation-models-understand-intuitive-physics]

models: LTX-Video (diffusion-based video generator) · method: Linear probing

JEPA (Joint Embedding Predictive Architecture)

The Invisible Hand of Physics: When Video Diffusion Models Know More Than They Show (2026)measured

Inverting video diffusion transformers makes physical plausibility linearly decodable

Details

Esmati et al. approximately invert the deterministic diffusion sampling of WAN-1.3B, CogVideoX-2B, and LTX-2B to recover intermediate states and attention maps [esmati-etal-2026-the-invisible-hand-of-physics] Physical plausibility is linearly decodable from the recovered states at ~81.27% average accuracy across IntPhys and InfLevel [esmati-etal-2026-the-invisible-hand-of-physics] This exceeds V-JEPA2 ViT-L and VideoMAE-Large baselines probed on their own forward-pass activations, so diffusion models implicitly encode physical structure recoverable only via inversion [esmati-etal-2026-the-invisible-hand-of-physics]

models: V-JEPA (video joint-embedding predictive architecture) · method: Diffusion-inversion probing, Linear probing
Do Video Foundation Models Understand Intuitive Physics? A Layerwise Probing Analysis (2026)measured

Video foundation models vary in physics decodability, V-JEPA strongest

Details

Punzo et al. fit frozen-feature linear probes layer-by-layer on V-JEPA, VideoMAE, and LTX-Video, evaluated on the IntPhys2 and Minimal Video Pairs intuitive-physics benchmarks [punzo-etal-2026-do-video-foundation-models-understand-intuitive-physics] V-JEPA (ViT-H) achieves the strongest results, especially with temporal-dynamics probes, VideoMAE is competitive, and LTX-Video recovers weaker but above-chance signal [punzo-etal-2026-do-video-foundation-models-understand-intuitive-physics] Physics-relevant information is weakest in early layers and peaks at intermediate-to-late depth [punzo-etal-2026-do-video-foundation-models-understand-intuitive-physics] Frame-order shuffling substantially degrades performance, confirming the signal reflects genuine temporal/physical structure rather than static appearance [punzo-etal-2026-do-video-foundation-models-understand-intuitive-physics]

models: V-JEPA (video joint-embedding predictive architecture) · method: Linear probing

VideoMAE

The Invisible Hand of Physics: When Video Diffusion Models Know More Than They Show (2026)measured

Inverting video diffusion transformers makes physical plausibility linearly decodable

Details

Esmati et al. approximately invert the deterministic diffusion sampling of WAN-1.3B, CogVideoX-2B, and LTX-2B to recover intermediate states and attention maps [esmati-etal-2026-the-invisible-hand-of-physics] Physical plausibility is linearly decodable from the recovered states at ~81.27% average accuracy across IntPhys and InfLevel [esmati-etal-2026-the-invisible-hand-of-physics] This exceeds V-JEPA2 ViT-L and VideoMAE-Large baselines probed on their own forward-pass activations, so diffusion models implicitly encode physical structure recoverable only via inversion [esmati-etal-2026-the-invisible-hand-of-physics]

models: VideoMAE-base · method: Diffusion-inversion probing, Linear probing
Do Video Foundation Models Understand Intuitive Physics? A Layerwise Probing Analysis (2026)measured

Video foundation models vary in physics decodability, V-JEPA strongest

Details

Punzo et al. fit frozen-feature linear probes layer-by-layer on V-JEPA, VideoMAE, and LTX-Video, evaluated on the IntPhys2 and Minimal Video Pairs intuitive-physics benchmarks [punzo-etal-2026-do-video-foundation-models-understand-intuitive-physics] V-JEPA (ViT-H) achieves the strongest results, especially with temporal-dynamics probes, VideoMAE is competitive, and LTX-Video recovers weaker but above-chance signal [punzo-etal-2026-do-video-foundation-models-understand-intuitive-physics] Physics-relevant information is weakest in early layers and peaks at intermediate-to-late depth [punzo-etal-2026-do-video-foundation-models-understand-intuitive-physics] Frame-order shuffling substantially degrades performance, confirming the signal reflects genuine temporal/physical structure rather than static appearance [punzo-etal-2026-do-video-foundation-models-understand-intuitive-physics]

models: VideoMAE-base · method: Linear probing

BERT

DirectProbe: Studying Representations without Classifiers (2021)measured

DirectProbe finds convex label clusters whose geometry predicts accuracy

Details

Zhou & Srikumar introduce DirectProbe, agglomerative clustering that merges same-label points only while their convex hulls stay disjoint from other labels, testing linear separability without training a classifier [zhou-srikumar-2021] Across BERT, RoBERTa, and ELMo on five linguistic tasks, the number of convex clusters matches the label count exactly when the representation is linearly separable (e.g. 17 clusters for 17-label POS) [zhou-srikumar-2021] The minimum inter-cluster margin tracks best achievable classifier accuracy across layers and fine-tuning, and a training-free nearest-centroid pseudo-classifier correlates with true accuracy [zhou-srikumar-2021] Classifier errors concentrate on label pairs whose clusters sit at small geometric distance (97.17% of one task's errors in the smallest-distance bin) [zhou-srikumar-2021]

models: BERT-base-cased, BERT-large-cased · method: Convex-hull clustering (DirectProbe)
Emergence of Separable Manifolds in Deep Language Representations (2020)measured

Linguistic category manifolds untangle across transformer layers

Details

Mamou et al. treat each linguistic-label point cloud as an object manifold and use replica mean-field theory (MFTMA) across BERT, RoBERTa, ALBERT, DistilBERT, and GPT-1 [mamou-etal-2020] On masked-token input, manifold capacity (linear separability) consistently increases across layers, driven by joint reduction in manifold radius, dimension, and inter-manifold center correlation [mamou-etal-2020] On unmasked input the effect reverses for word identity while higher-level categories (POS, NER, semantic tags) still gain separability, most strongly for part-of-speech-ambiguous words [mamou-etal-2020]

models: BERT-base-cased · method: Mean-Field Theoretic Manifold Analysis (MFTMA)
A Closer Look at How Fine-tuning Changes BERT (2021)measured

Fine-tuning widens convex-cluster margins without rearranging the geometry

Details

Zhou & Srikumar reuse DirectProbe on five BERT checkpoints across four linearly-separable tasks to test what fine-tuning does to convex-cluster geometry [zhou-srikumar-2021-finetuning] Fine-tuning consistently increases the minimum inter-cluster margin between every label pair, the geometric mechanism behind accuracy gains [zhou-srikumar-2021-finetuning] A new Spatial Similarity metric shows this is not arbitrary rearrangement: pre/post cluster geometry retains Pearson correlation above 0.5 in higher layers [zhou-srikumar-2021-finetuning] The same metric explains the one exception (BERT-small preposition-supersense accuracy dropping) via the lowest train/test spatial similarity of any condition (0.44) [zhou-srikumar-2021-finetuning]

models: BERT-tiny, BERT-mini, BERT-small, BERT-medium, BERT-base-uncased · method: Convex-hull clustering (DirectProbe), Spatial similarity (inter-cluster distance correlation)
Finding Lexical Identity and Inflectional Morphology in Modern Language Models (2025)measured

Morphology stays linearly separable while lexical identity turns nonlinear with depth

Details

Li & Subramani train ridge-regression linear probes and nonlinear MLP probes across 25 pretrained models to predict word lemma and inflectional features [li-subramani-2025-finding-lexical-identity-and-inflectional-morphology-in-modern-language-models] Lemma is linearly separable in early layers (0.8-1.0) but linear accuracy declines toward final layers while MLP accuracy declines much less, so lexical identity becomes increasingly nonlinearly encoded [li-subramani-2025-finding-lexical-identity-and-inflectional-morphology-in-modern-language-models] Inflectional morphology stays linearly separable at 0.9-1.0 uniformly across every layer, with MLP probes adding no benefit [li-subramani-2025-finding-lexical-identity-and-inflectional-morphology-in-modern-language-models] Control-task selectivity confirms inflection is a generalizable linear abstraction (selectivity 0.4-0.6) while lemma decodability reflects near-zero-selectivity memorization [li-subramani-2025-finding-lexical-identity-and-inflectional-morphology-in-modern-language-models]

models: BERT-base-uncased, BERT-large-uncased · method: Linear probing, Mlp probing
Outlier Dimensions Encode Task-Specific Knowledge (2023)measured

Outlier dimensions persist and can solve GLUE tasks

Details

- Outlier dimensions (variance >=5x the average) that arise in pretraining persist as outliers after fine-tuning regardless of task or seed (GPT-2's exact top-3 persist). [rudman-etal-2023-outlier-dimensions-encode-task-specific-knowledge] - A single principal outlier dimension with a learned threshold can complete GLUE binary-classification tasks, with model-dependent accuracy loss: low for GPT-2 (2.85% avg) and ALBERT (6.12%) but high for BERT (16.33%), RoBERTa (19.23%) and DistilBERT (19.12%); Pythia 70M/160M/410M lose 12.58/7.48/11.44%. [rudman-etal-2023-outlier-dimensions-encode-task-specific-knowledge] - The often-quoted "0-3.4%" figures are QNLI-only (ALBERT 3.42%, Pythia-410M 0.0%), not the cross-task averages; Pythia-410M (11.44% avg) is not low-loss. [rudman-etal-2023-outlier-dimensions-encode-task-specific-knowledge] - Tested on eight encoder and decoder models (BERT, ALBERT, DistilBERT, RoBERTa, GPT-2, Pythia-70M/160M/410M); the paper names families/sizes, not exact checkpoints. [rudman-etal-2023-outlier-dimensions-encode-task-specific-knowledge]

models: BERT-base-uncased · method: Rogue-dimension analysis

RoBERTa

DirectProbe: Studying Representations without Classifiers (2021)measured

DirectProbe finds convex label clusters whose geometry predicts accuracy

Details

Zhou & Srikumar introduce DirectProbe, agglomerative clustering that merges same-label points only while their convex hulls stay disjoint from other labels, testing linear separability without training a classifier [zhou-srikumar-2021] Across BERT, RoBERTa, and ELMo on five linguistic tasks, the number of convex clusters matches the label count exactly when the representation is linearly separable (e.g. 17 clusters for 17-label POS) [zhou-srikumar-2021] The minimum inter-cluster margin tracks best achievable classifier accuracy across layers and fine-tuning, and a training-free nearest-centroid pseudo-classifier correlates with true accuracy [zhou-srikumar-2021] Classifier errors concentrate on label pairs whose clusters sit at small geometric distance (97.17% of one task's errors in the smallest-distance bin) [zhou-srikumar-2021]

models: RoBERTa-base, RoBERTa-large · method: Convex-hull clustering (DirectProbe)
Emergence of Separable Manifolds in Deep Language Representations (2020)measured

Linguistic category manifolds untangle across transformer layers

Details

Mamou et al. treat each linguistic-label point cloud as an object manifold and use replica mean-field theory (MFTMA) across BERT, RoBERTa, ALBERT, DistilBERT, and GPT-1 [mamou-etal-2020] On masked-token input, manifold capacity (linear separability) consistently increases across layers, driven by joint reduction in manifold radius, dimension, and inter-manifold center correlation [mamou-etal-2020] On unmasked input the effect reverses for word identity while higher-level categories (POS, NER, semantic tags) still gain separability, most strongly for part-of-speech-ambiguous words [mamou-etal-2020]

models: RoBERTa-base · method: Mean-Field Theoretic Manifold Analysis (MFTMA)
Outlier Dimensions Encode Task-Specific Knowledge (2023)measured

Outlier dimensions persist and can solve GLUE tasks

Details

- Outlier dimensions (variance >=5x the average) that arise in pretraining persist as outliers after fine-tuning regardless of task or seed (GPT-2's exact top-3 persist). [rudman-etal-2023-outlier-dimensions-encode-task-specific-knowledge] - A single principal outlier dimension with a learned threshold can complete GLUE binary-classification tasks, with model-dependent accuracy loss: low for GPT-2 (2.85% avg) and ALBERT (6.12%) but high for BERT (16.33%), RoBERTa (19.23%) and DistilBERT (19.12%); Pythia 70M/160M/410M lose 12.58/7.48/11.44%. [rudman-etal-2023-outlier-dimensions-encode-task-specific-knowledge] - The often-quoted "0-3.4%" figures are QNLI-only (ALBERT 3.42%, Pythia-410M 0.0%), not the cross-task averages; Pythia-410M (11.44% avg) is not low-loss. [rudman-etal-2023-outlier-dimensions-encode-task-specific-knowledge] - Tested on eight encoder and decoder models (BERT, ALBERT, DistilBERT, RoBERTa, GPT-2, Pythia-70M/160M/410M); the paper names families/sizes, not exact checkpoints. [rudman-etal-2023-outlier-dimensions-encode-task-specific-knowledge]

models: RoBERTa-base · method: Rogue-dimension analysis

ELMo

DirectProbe: Studying Representations without Classifiers (2021)measured

DirectProbe finds convex label clusters whose geometry predicts accuracy

Details

Zhou & Srikumar introduce DirectProbe, agglomerative clustering that merges same-label points only while their convex hulls stay disjoint from other labels, testing linear separability without training a classifier [zhou-srikumar-2021] Across BERT, RoBERTa, and ELMo on five linguistic tasks, the number of convex clusters matches the label count exactly when the representation is linearly separable (e.g. 17 clusters for 17-label POS) [zhou-srikumar-2021] The minimum inter-cluster margin tracks best achievable classifier accuracy across layers and fine-tuning, and a training-free nearest-centroid pseudo-classifier correlates with true accuracy [zhou-srikumar-2021] Classifier errors concentrate on label pairs whose clusters sit at small geometric distance (97.17% of one task's errors in the smallest-distance bin) [zhou-srikumar-2021]

models: ELMo (5.5B-word pretrained, 2-layer biLSTM) · method: Convex-hull clustering (DirectProbe)

SchNet (continuous-filter convolutional GNN for molecules)

Global geometry of chemical graph neural network representations in terms of chemical moieties (2024)measured

A real SchNet-family GNN's 128-parameter QM9 molecular embedding space reduces to around 5 effective parameters, with sharp linear boundaries separating chemical moieties

Details

Dimension reduction and linear discriminant analysis applied to the hidden-layer embedding vectors of a real SchNet-family GNN trained on the real QM9 molecular dataset show the fully-trained 128-parameter embedding space reduces to a low parametric space of around 5 important parameters [el-samman-etal-2024-global-geometry-chemical-gnn-moieties] Sharp linear boundaries separate chemical moieties within this reduced embedding space, with classification error below 5x10^-4 [el-samman-etal-2024-global-geometry-chemical-gnn-moieties] Euclidean distance in the embedding space functions as a molecular similarity measure competitive with the hand-engineered SOAP descriptor, and embedding coordinates linearly predict pKa and NMR chemical-shift observables [el-samman-etal-2024-global-geometry-chemical-gnn-moieties]

models: SchNet-family GNN, trained on QM9 (El-Samman, Husain, Huynh, De Castro, Morton & De Baerdemacker) · method:

ALBERT

Emergence of Separable Manifolds in Deep Language Representations (2020)measured

Linguistic category manifolds untangle across transformer layers

Details

Mamou et al. treat each linguistic-label point cloud as an object manifold and use replica mean-field theory (MFTMA) across BERT, RoBERTa, ALBERT, DistilBERT, and GPT-1 [mamou-etal-2020] On masked-token input, manifold capacity (linear separability) consistently increases across layers, driven by joint reduction in manifold radius, dimension, and inter-manifold center correlation [mamou-etal-2020] On unmasked input the effect reverses for word identity while higher-level categories (POS, NER, semantic tags) still gain separability, most strongly for part-of-speech-ambiguous words [mamou-etal-2020]

models: ALBERT-base-v1 · method: Mean-Field Theoretic Manifold Analysis (MFTMA)
Outlier Dimensions Encode Task-Specific Knowledge (2023)measured

Outlier dimensions persist and can solve GLUE tasks

Details

- Outlier dimensions (variance >=5x the average) that arise in pretraining persist as outliers after fine-tuning regardless of task or seed (GPT-2's exact top-3 persist). [rudman-etal-2023-outlier-dimensions-encode-task-specific-knowledge] - A single principal outlier dimension with a learned threshold can complete GLUE binary-classification tasks, with model-dependent accuracy loss: low for GPT-2 (2.85% avg) and ALBERT (6.12%) but high for BERT (16.33%), RoBERTa (19.23%) and DistilBERT (19.12%); Pythia 70M/160M/410M lose 12.58/7.48/11.44%. [rudman-etal-2023-outlier-dimensions-encode-task-specific-knowledge] - The often-quoted "0-3.4%" figures are QNLI-only (ALBERT 3.42%, Pythia-410M 0.0%), not the cross-task averages; Pythia-410M (11.44% avg) is not low-loss. [rudman-etal-2023-outlier-dimensions-encode-task-specific-knowledge] - Tested on eight encoder and decoder models (BERT, ALBERT, DistilBERT, RoBERTa, GPT-2, Pythia-70M/160M/410M); the paper names families/sizes, not exact checkpoints. [rudman-etal-2023-outlier-dimensions-encode-task-specific-knowledge]

models: ALBERT-base-v1 · method: Rogue-dimension analysis

DistilBERT

Emergence of Separable Manifolds in Deep Language Representations (2020)measured

Linguistic category manifolds untangle across transformer layers

Details

Mamou et al. treat each linguistic-label point cloud as an object manifold and use replica mean-field theory (MFTMA) across BERT, RoBERTa, ALBERT, DistilBERT, and GPT-1 [mamou-etal-2020] On masked-token input, manifold capacity (linear separability) consistently increases across layers, driven by joint reduction in manifold radius, dimension, and inter-manifold center correlation [mamou-etal-2020] On unmasked input the effect reverses for word identity while higher-level categories (POS, NER, semantic tags) still gain separability, most strongly for part-of-speech-ambiguous words [mamou-etal-2020]

models: DistilBERT-base-uncased · method: Mean-Field Theoretic Manifold Analysis (MFTMA)
Outlier Dimensions Encode Task-Specific Knowledge (2023)measured

Outlier dimensions persist and can solve GLUE tasks

Details

- Outlier dimensions (variance >=5x the average) that arise in pretraining persist as outliers after fine-tuning regardless of task or seed (GPT-2's exact top-3 persist). [rudman-etal-2023-outlier-dimensions-encode-task-specific-knowledge] - A single principal outlier dimension with a learned threshold can complete GLUE binary-classification tasks, with model-dependent accuracy loss: low for GPT-2 (2.85% avg) and ALBERT (6.12%) but high for BERT (16.33%), RoBERTa (19.23%) and DistilBERT (19.12%); Pythia 70M/160M/410M lose 12.58/7.48/11.44%. [rudman-etal-2023-outlier-dimensions-encode-task-specific-knowledge] - The often-quoted "0-3.4%" figures are QNLI-only (ALBERT 3.42%, Pythia-410M 0.0%), not the cross-task averages; Pythia-410M (11.44% avg) is not low-loss. [rudman-etal-2023-outlier-dimensions-encode-task-specific-knowledge] - Tested on eight encoder and decoder models (BERT, ALBERT, DistilBERT, RoBERTa, GPT-2, Pythia-70M/160M/410M); the paper names families/sizes, not exact checkpoints. [rudman-etal-2023-outlier-dimensions-encode-task-specific-knowledge]

models: DistilBERT-base-uncased · method: Rogue-dimension analysis

GPT

Emergence of Separable Manifolds in Deep Language Representations (2020)measured

Linguistic category manifolds untangle across transformer layers

Details

Mamou et al. treat each linguistic-label point cloud as an object manifold and use replica mean-field theory (MFTMA) across BERT, RoBERTa, ALBERT, DistilBERT, and GPT-1 [mamou-etal-2020] On masked-token input, manifold capacity (linear separability) consistently increases across layers, driven by joint reduction in manifold radius, dimension, and inter-manifold center correlation [mamou-etal-2020] On unmasked input the effect reverses for word identity while higher-level categories (POS, NER, semantic tags) still gain separability, most strongly for part-of-speech-ambiguous words [mamou-etal-2020]

models: GPT-1 (OpenAI GPT) · method: Mean-Field Theoretic Manifold Analysis (MFTMA)
Finding Lexical Identity and Inflectional Morphology in Modern Language Models (2025)measured

Morphology stays linearly separable while lexical identity turns nonlinear with depth

Details

Li & Subramani train ridge-regression linear probes and nonlinear MLP probes across 25 pretrained models to predict word lemma and inflectional features [li-subramani-2025-finding-lexical-identity-and-inflectional-morphology-in-modern-language-models] Lemma is linearly separable in early layers (0.8-1.0) but linear accuracy declines toward final layers while MLP accuracy declines much less, so lexical identity becomes increasingly nonlinearly encoded [li-subramani-2025-finding-lexical-identity-and-inflectional-morphology-in-modern-language-models] Inflectional morphology stays linearly separable at 0.9-1.0 uniformly across every layer, with MLP probes adding no benefit [li-subramani-2025-finding-lexical-identity-and-inflectional-morphology-in-modern-language-models] Control-task selectivity confirms inflection is a generalizable linear abstraction (selectivity 0.4-0.6) while lemma decodability reflects near-zero-selectivity memorization [li-subramani-2025-finding-lexical-identity-and-inflectional-morphology-in-modern-language-models]

models: GPT-2-small, GPT-2-Large, GPT-2-XL · method: Linear probing, Mlp probing
What Really Controls Temporal Reasoning in LLMs: Tokenisation or Representation of Time? (2026)measured

Temporal-probe decodability predicts reasoning in high-resource languages

Details

Bhatia et al. introduce MultiTempBench and evaluate 20 LLMs, combining a tokenization-fragmentation metric with linear-regression probes decoding Year/Month/Day (probe R^2 as temporal linearity) [bhatia-etal-2026-what-really-controls-temporal-reasoning-in-llms] A crossed mixed-effects regression over 285,000 predictions finds temporal linearity is the strongest predictor of reasoning accuracy in high-resource languages (English r=0.77) [bhatia-etal-2026-what-really-controls-temporal-reasoning-in-llms] Tokenization fragmentation dominates in low-resource languages (Hausa r=-0.97); the finding is purely correlational [bhatia-etal-2026-what-really-controls-temporal-reasoning-in-llms]

models: GPT-4o · method: Linear probing

Whisper

Analyzing Language and Geographical Variation in Speech Representations Across 60 Indic Languages (2026)measured

Indic dialect geography nests linearly inside each language's subspace

Details

Kumar J et al. fine-tune Whisper-base and Wav2Vec2.0-base on 60 Indic languages/165 districts under joint versus language-only supervision [kumarj-etal-2026-analyzing-language-and-geographical-variation-in-speech-representations-across-60-indic-languages] Language-conditioned logistic-regression probes decode district structure within each language's subspace (Whisper-base mean F1 91.29 joint vs 80.89 language-only; Wav2Vec2 87.14 vs 77.05) [kumarj-etal-2026-analyzing-language-and-geographical-variation-in-speech-representations-across-60-indic-languages] A normalized conditional mutual-information analysis reveals an asymmetric hierarchical geometry (AUC_D|L=2.68 vs AUC_L|D=5.43) with district clusters nested inside language structure [kumarj-etal-2026-analyzing-language-and-geographical-variation-in-speech-representations-across-60-indic-languages]

models: Whisper base · method: Linear probing

wav2vec 2.0

Analyzing Language and Geographical Variation in Speech Representations Across 60 Indic Languages (2026)measured

Indic dialect geography nests linearly inside each language's subspace

Details

Kumar J et al. fine-tune Whisper-base and Wav2Vec2.0-base on 60 Indic languages/165 districts under joint versus language-only supervision [kumarj-etal-2026-analyzing-language-and-geographical-variation-in-speech-representations-across-60-indic-languages] Language-conditioned logistic-regression probes decode district structure within each language's subspace (Whisper-base mean F1 91.29 joint vs 80.89 language-only; Wav2Vec2 87.14 vs 77.05) [kumarj-etal-2026-analyzing-language-and-geographical-variation-in-speech-representations-across-60-indic-languages] A normalized conditional mutual-information analysis reveals an asymmetric hierarchical geometry (AUC_D|L=2.68 vs AUC_L|D=5.43) with district clusters nested inside language structure [kumarj-etal-2026-analyzing-language-and-geographical-variation-in-speech-representations-across-60-indic-languages]

models: Wav2Vec 2.0 Base (LibriSpeech-960h) · method: Linear probing

Gaperon

Language-Switching Triggers Take a Latent Detour Through Language Models (2026)measured

A backdoor trigger hides orthogonal to the probed language direction

Details

Kulumba et al. plant a 9-token Latin trigger during pretraining of Gaperon-8B that switches generation from English to French, then decompose a 3-phase causal circuit via activation patching [kulumba-etal-2026-language-switching-triggers-take-a-latent-detour-through-language-models] In the critical phase, linear language-identity probes classify the representation as English throughout mid-to-late layers even though patching confirms the trigger is causally present [kulumba-etal-2026-language-switching-triggers-take-a-latent-detour-through-language-models] The trigger occupies a subspace orthogonal to the natural language-identity direction, converted to French only by a final-layer MLP that accounts for approximately 62% (plus or minus 8%) of the total causal effect [kulumba-etal-2026-language-switching-triggers-take-a-latent-detour-through-language-models]

models: Gaperon-8B · method: Activation patching, Linear probing

Pythia

Finding Lexical Identity and Inflectional Morphology in Modern Language Models (2025)measured

Morphology stays linearly separable while lexical identity turns nonlinear with depth

Details

Li & Subramani train ridge-regression linear probes and nonlinear MLP probes across 25 pretrained models to predict word lemma and inflectional features [li-subramani-2025-finding-lexical-identity-and-inflectional-morphology-in-modern-language-models] Lemma is linearly separable in early layers (0.8-1.0) but linear accuracy declines toward final layers while MLP accuracy declines much less, so lexical identity becomes increasingly nonlinearly encoded [li-subramani-2025-finding-lexical-identity-and-inflectional-morphology-in-modern-language-models] Inflectional morphology stays linearly separable at 0.9-1.0 uniformly across every layer, with MLP probes adding no benefit [li-subramani-2025-finding-lexical-identity-and-inflectional-morphology-in-modern-language-models] Control-task selectivity confirms inflection is a generalizable linear abstraction (selectivity 0.4-0.6) while lemma decodability reflects near-zero-selectivity memorization [li-subramani-2025-finding-lexical-identity-and-inflectional-morphology-in-modern-language-models]

models: Pythia-6.9B · method: Linear probing, Mlp probing
Outlier Dimensions Encode Task-Specific Knowledge (2023)measured

Outlier dimensions persist and can solve GLUE tasks

Details

- Outlier dimensions (variance >=5x the average) that arise in pretraining persist as outliers after fine-tuning regardless of task or seed (GPT-2's exact top-3 persist). [rudman-etal-2023-outlier-dimensions-encode-task-specific-knowledge] - A single principal outlier dimension with a learned threshold can complete GLUE binary-classification tasks, with model-dependent accuracy loss: low for GPT-2 (2.85% avg) and ALBERT (6.12%) but high for BERT (16.33%), RoBERTa (19.23%) and DistilBERT (19.12%); Pythia 70M/160M/410M lose 12.58/7.48/11.44%. [rudman-etal-2023-outlier-dimensions-encode-task-specific-knowledge] - The often-quoted "0-3.4%" figures are QNLI-only (ALBERT 3.42%, Pythia-410M 0.0%), not the cross-task averages; Pythia-410M (11.44% avg) is not low-loss. [rudman-etal-2023-outlier-dimensions-encode-task-specific-knowledge] - Tested on eight encoder and decoder models (BERT, ALBERT, DistilBERT, RoBERTa, GPT-2, Pythia-70M/160M/410M); the paper names families/sizes, not exact checkpoints. [rudman-etal-2023-outlier-dimensions-encode-task-specific-knowledge]

models: Pythia-70M, Pythia-160M, Pythia-410M · method: Rogue-dimension analysis

OLMo 2

Finding Lexical Identity and Inflectional Morphology in Modern Language Models (2025)measured

Morphology stays linearly separable while lexical identity turns nonlinear with depth

Details

Li & Subramani train ridge-regression linear probes and nonlinear MLP probes across 25 pretrained models to predict word lemma and inflectional features [li-subramani-2025-finding-lexical-identity-and-inflectional-morphology-in-modern-language-models] Lemma is linearly separable in early layers (0.8-1.0) but linear accuracy declines toward final layers while MLP accuracy declines much less, so lexical identity becomes increasingly nonlinearly encoded [li-subramani-2025-finding-lexical-identity-and-inflectional-morphology-in-modern-language-models] Inflectional morphology stays linearly separable at 0.9-1.0 uniformly across every layer, with MLP probes adding no benefit [li-subramani-2025-finding-lexical-identity-and-inflectional-morphology-in-modern-language-models] Control-task selectivity confirms inflection is a generalizable linear abstraction (selectivity 0.4-0.6) while lemma decodability reflects near-zero-selectivity memorization [li-subramani-2025-finding-lexical-identity-and-inflectional-morphology-in-modern-language-models]

models: OLMo 2 7B · method: Linear probing, Mlp probing

Ovis

Probing Visual Concepts in Lightweight Vision-Language Models for Automated Driving (2026)measured

Lightweight VLMs linearly encode object presence but never orientation

Details

Theodoridis et al. use CARLA counterfactual pairs and layer-wise linear probes across four lightweight VLMs (Ovis2.5-2B, InternVL3.5-2B, VST-3B SFT and RL) [theodoridis-etal-2026-probing-visual-concepts-in-lightweight-vision-language-models-for-automated-driving] Object/agent presence is linearly encoded from mid-vision-encoder through final LLM layers, while orientation is never linearly encoded anywhere, and increasing object distance rapidly degrades separability [theodoridis-etal-2026-probing-visual-concepts-in-lightweight-vision-language-models-for-automated-driving] Two failure modes are dissociated: perceptual (concept not encoded) versus cognitive (concept present but the model still answers wrong) [theodoridis-etal-2026-probing-visual-concepts-in-lightweight-vision-language-models-for-automated-driving] Activation steering along the probe directions causally confirms they drive behavior rather than being correlational [theodoridis-etal-2026-probing-visual-concepts-in-lightweight-vision-language-models-for-automated-driving]

models: Ovis2.5-2B · method: Supervised direction search (gradient-based optimization), Causal interventions (steering)

VST

Probing Visual Concepts in Lightweight Vision-Language Models for Automated Driving (2026)measured

Lightweight VLMs linearly encode object presence but never orientation

Details

Theodoridis et al. use CARLA counterfactual pairs and layer-wise linear probes across four lightweight VLMs (Ovis2.5-2B, InternVL3.5-2B, VST-3B SFT and RL) [theodoridis-etal-2026-probing-visual-concepts-in-lightweight-vision-language-models-for-automated-driving] Object/agent presence is linearly encoded from mid-vision-encoder through final LLM layers, while orientation is never linearly encoded anywhere, and increasing object distance rapidly degrades separability [theodoridis-etal-2026-probing-visual-concepts-in-lightweight-vision-language-models-for-automated-driving] Two failure modes are dissociated: perceptual (concept not encoded) versus cognitive (concept present but the model still answers wrong) [theodoridis-etal-2026-probing-visual-concepts-in-lightweight-vision-language-models-for-automated-driving] Activation steering along the probe directions causally confirms they drive behavior rather than being correlational [theodoridis-etal-2026-probing-visual-concepts-in-lightweight-vision-language-models-for-automated-driving]

models: VST-3B · method: Supervised direction search (gradient-based optimization), Causal interventions (steering)

DeepSeek

What Really Controls Temporal Reasoning in LLMs: Tokenisation or Representation of Time? (2026)measured

Temporal-probe decodability predicts reasoning in high-resource languages

Details

Bhatia et al. introduce MultiTempBench and evaluate 20 LLMs, combining a tokenization-fragmentation metric with linear-regression probes decoding Year/Month/Day (probe R^2 as temporal linearity) [bhatia-etal-2026-what-really-controls-temporal-reasoning-in-llms] A crossed mixed-effects regression over 285,000 predictions finds temporal linearity is the strongest predictor of reasoning accuracy in high-resource languages (English r=0.77) [bhatia-etal-2026-what-really-controls-temporal-reasoning-in-llms] Tokenization fragmentation dominates in low-resource languages (Hausa r=-0.97); the finding is purely correlational [bhatia-etal-2026-what-really-controls-temporal-reasoning-in-llms]

models: DeepSeek-R1-Distill-Qwen-7B · method: Linear probing
Reasoning Models Know When They're Right: Probing Hidden States for Self-Verification (2025)measured

A probe reads answer correctness before reasoning models state it

Details

Zhang et al. fit probes (many converging to purely linear) on hidden states at intermediate-answer positions across six reasoning models [zhang-etal-2025-reasoning-models-know-when-theyre-right-probing-hidden-states-for-self-verification] Correctness is predicted at ROC-AUC above 0.7 with calibration error under 0.1 in every model, including for future not-yet-stated answers later in the trace [zhang-etal-2025-reasoning-models-know-when-theyre-right-probing-hidden-states-for-self-verification] The effect is much weaker in a non-reasoning baseline (Llama-3.1-8B-Instruct), tying the signal to reasoning-specific training [zhang-etal-2025-reasoning-models-know-when-theyre-right-probing-hidden-states-for-self-verification] Used as an early-exit verifier, the probe cuts inference tokens by 24% with no accuracy loss [zhang-etal-2025-reasoning-models-know-when-theyre-right-probing-hidden-states-for-self-verification]

models: DeepSeek-R1-Distill-Llama-8B, DeepSeek-R1-Distill-Llama-70B, DeepSeek-R1-Distill-Qwen-1.5B, DeepSeek-R1-Distill-Qwen-7B, DeepSeek-R1-Distill-Qwen-32B · method: Linear probing, Mlp probing
LLM Reasoning as Trajectories: Step-Specific Representation Geometry and Correctness Signals (2026)measured

Chain-of-thought traverses step-specific subspaces that sharpen with depth

Details

Sun et al. train per-layer per-step one-vs-rest logistic probes on activations preceding each reasoning-step marker in Llama-3.1-8B Base, Instruct, and R1-Distill [sun-etal-2026-llm-reasoning-as-trajectories] Later-step separability starts low at the embedding layer and exceeds 0.90 only after several layers, while early steps are separable from layer 0; shuffling controls collapse to about 0.59 [sun-etal-2026-llm-reasoning-as-trajectories] Probes transfer across training regimes, prompt formats, and datasets (0.83-1.00), and late-trajectory PCA features predict eventual correctness mid-generation (ROC-AUC up to 0.87) [sun-etal-2026-llm-reasoning-as-trajectories] A termination-direction steering vector causally controls reasoning length, and a rank-32 trajectory-correction update raises 6-7-step GSM8K accuracy by about 7.6 points while preserving correct answers [sun-etal-2026-llm-reasoning-as-trajectories]

models: DeepSeek-R1-Distill-Llama-8B · method: Linear probing, Causal interventions (steering)
Language Models Represent Beliefs of Self and Others (2024)measured

A mid-layer head group decodes another agent's beliefs and steers theory-of-mind

Details

Zhu et al. prompt Mistral-7B-Instruct (and DeepSeek-LLM-7B-Chat) with third-person belief narratives and fit logistic-regression probes on attention-head activations [zhu-etal-2024-language-models-represent-beliefs-of-self-and-others] A group of middle-layer attention heads decodes another agent's belief status at over 80% accuracy, distinct from the model's own self-belief representation [zhu-etal-2024-language-models-represent-beliefs-of-self-and-others] Steering these representations produces large changes in downstream Theory-of-Mind performance, generalizing across social-reasoning tasks [zhu-etal-2024-language-models-represent-beliefs-of-self-and-others]

models: DeepSeek-LLM-7B-Chat · method: Linear probing, Activation Steering (Addition)

Chemical Language Model (ChemBERTa / MoLFormer / RoBERTa-Zinc family)

Probing Chemical Language Models: Effects of Pre-training and Fine-tuning (2026)measured

Pretraining improves chemical-LM substructure awareness in upper layers

Details

Karnysheva, Klakow & Lee probe eight pretrained chemical language models and six randomly-initialized counterparts for 78 molecular substructures layer-by-layer [karnysheva-etal-2026-probing-chemical-language-models] Pretraining generally improves substructure awareness, particularly in upper layers, while randomly-initialized models already linearly encode ring structures at layer 1 [karnysheva-etal-2026-probing-chemical-language-models] Fine-tuning on two downstream tasks affects task-relevant substructures more than others [karnysheva-etal-2026-probing-chemical-language-models]

models: ChemBERTa-2 (77M molecules), MolFormer (100M molecules), RoBERTa-Zinc-480M · method: Linear probing

Vicuna

Unlocking the Future: Look-Ahead Planning Mechanistic Interpretability in LLMs (2024)measured

Linear probes decode future Blocksworld planning steps from one pass

Details

Men et al. train linear probes (versus a nonlinear control) on hidden states of Llama-2-7b-chat and Vicuna-7B fine-tuned on Blocksworld planning [men-etal-2024-unlocking-the-future-look-ahead-planning-mechanistic-interpretability-in-llms] A linear probe predicts the 6th-step-ahead decision from the 1st-step representation at 0.51 accuracy, decaying smoothly with prediction distance [men-etal-2024-unlocking-the-future-look-ahead-planning-mechanistic-interpretability-in-llms] Linear and nonlinear probes track the same decay, evidence the look-ahead information is linearly encoded; MHSA key-masking confirms which attention paths carry it [men-etal-2024-unlocking-the-future-look-ahead-planning-mechanistic-interpretability-in-llms]

models: Vicuna-7B · method: Linear probing, Causal interventions (steering)

Controlled Geospatial SSL Encoders (DINO/MAE/MoCo on SSL4EO Sentinel-1/2)

Probing Geospatial SSL Representations with Environmental Signals (2026)measured

Environmental variables never seen in pretraining are linearly decodable from frozen geospatial SSL encoders, with architecture-dependent strength

Details

Mocharla & Patel (2026) fit ridge-regression (linear) and MLP (nonlinear) probes for five ERA5 environmental variables (temperature, precipitation, solar radiation, pressure, soil water) -- never used during pretraining -- on real DINO/MAE/MoCo ViT-S/16 encoders trained identically on SSL4EO Sentinel-1/2 imagery, plus several public geospatial foundation models [mocharla-patel-2026-probing-geospatial-ssl-representations] DINO shows the strongest ERA5 decodability (linear R^2=0.50, MLP R^2=0.68) versus MAE/MoCo near-random despite similar downstream segmentation mIoU; linear-probe R^2 correlates more strongly with downstream agricultural-task utility than MLP R^2 (rho=0.75 vs 0.63) [mocharla-patel-2026-probing-geospatial-ssl-representations] Intrinsic geometry metrics (effective rank, uniformity) correlate with probe performance (rho~=0.40-0.45); purely associative, the authors explicitly state the analysis is not causal [mocharla-patel-2026-probing-geospatial-ssl-representations]

models: DINO ViT-S/16 trained on SSL4EO (Sentinel-1/2 satellite imagery) · method: Linear probing

AlexNet

Separability and Geometry of Object Manifolds in Deep Neural Networks (2020)measured

Object manifolds untangle across CNN depth, causally driven by geometry

Details

Cohen et al. apply replica mean-field manifold-capacity theory to ImageNet object manifolds in AlexNet, VGG-16, and ResNet-50 [cohen-etal-2020-separability-geometry-object-manifolds] Classification capacity increases substantially along the trained hierarchy, up to nearly two orders of magnitude over shuffled-label controls for high-variability smooth manifolds [cohen-etal-2020-separability-geometry-object-manifolds] Manifold dimension and radius shrink and inter-manifold center correlations decrease with depth, none of which occur in untrained or label-shuffled controls, isolating training as the cause [cohen-etal-2020-separability-geometry-object-manifolds] Three manifold-perturbation interventions (size scaling, ball approximation, center randomization) confirm dimension reduction drives 55-90% of the capacity gain [cohen-etal-2020-separability-geometry-object-manifolds]

models: AlexNet (ImageNet image classifier, supervised) · method: Mean-Field Theoretic Manifold Analysis (MFTMA)

VGG

Separability and Geometry of Object Manifolds in Deep Neural Networks (2020)measured

Object manifolds untangle across CNN depth, causally driven by geometry

Details

Cohen et al. apply replica mean-field manifold-capacity theory to ImageNet object manifolds in AlexNet, VGG-16, and ResNet-50 [cohen-etal-2020-separability-geometry-object-manifolds] Classification capacity increases substantially along the trained hierarchy, up to nearly two orders of magnitude over shuffled-label controls for high-variability smooth manifolds [cohen-etal-2020-separability-geometry-object-manifolds] Manifold dimension and radius shrink and inter-manifold center correlations decrease with depth, none of which occur in untrained or label-shuffled controls, isolating training as the cause [cohen-etal-2020-separability-geometry-object-manifolds] Three manifold-perturbation interventions (size scaling, ball approximation, center randomization) confirm dimension reduction drives 55-90% of the capacity gain [cohen-etal-2020-separability-geometry-object-manifolds]

models: VGG (image classifier, various depths) · method: Mean-Field Theoretic Manifold Analysis (MFTMA)

AION-1

AION-1: Omnimodal Foundation Model for Astronomical Sciences (2025)measured

A frozen linear morphology classifier on AION-1 embeddings transfers zero-shot across telescope surveys

Details

Parker, Lanusse, Shen et al. (2025) train a 10-way linear/softmax morphology classifier (Galaxy Zoo-10 classes) on frozen AION-1 embeddings of Legacy Survey galaxy images, encoder unmodified [parker-etal-2025-aion-1-omnimodal-foundation-model-astronomical-sciences] Applying the identical, un-fine-tuned head to Hyper Suprime-Cam images from a different telescope/filter set/pixel scale (overlapping galaxies removed to prevent leakage) yields zero-shot accuracy (84.15-85.91%) matching or exceeding native-domain training accuracy (83.95-87.16%) across three model sizes [parker-etal-2025-aion-1-omnimodal-foundation-model-astronomical-sciences] No causal intervention is performed; the paper is otherwise benchmark-dominated, with this cross-survey linear transfer the one genuine emergent-structure finding [parker-etal-2025-aion-1-omnimodal-foundation-model-astronomical-sciences]

models: AION-1 (omnimodal foundation model for astronomical sciences) · method: Linear probing

GNN diagnostic-classifier probing suite (GCN/GAT/GIN on Grid-House and ClinTox)

Do Graph Neural Network States Contain Graph Properties? (2024)measured

Graph properties decode linearly from GNN embeddings, collapsing with depth

Details

Pelletreau-Duris et al. fit linear-regression probes mapping GCN/GAT/GIN activations to dozens of graph-theoretic properties at node and graph level [pelletreau-duris-etal-2024-do-graph-neural-network-states-contain-graph-properties] Size-correlated global properties decode near-perfectly early (R^2=1.00) but collapse to about 0.04-0.07 by the final task-specialized layer, while node betweenness centrality stays roughly flat [pelletreau-duris-etal-2024-do-graph-neural-network-states-contain-graph-properties] A 0.992 correlation between task accuracy and maximum probing score links how well embeddings encode graph structure to downstream performance [pelletreau-duris-etal-2024-do-graph-neural-network-states-contain-graph-properties]

models: GCN / GAT / GIN (trained on synthetic Grid-House and ClinTox molecular graphs) · method: Linear probing

HuBERT

Training-Free Cross-Lingual Dysarthria Severity Assessment via Phonological Subspace Analysis in Self-Supervised Speech Representations (2026)measured

Phonological contrast directions in frozen HuBERT-base embeddings collapse in separability with dysarthria severity, training-free and across languages

Details

For each of 12 phonological contrasts (nasality, voicing, stridency, sonorance, manner, 4 vowel features), a difference-of-means direction is computed in 768-dimensional layer-12 HuBERT-base activations and separability along that direction is measured via d-prime, with no training on pathological data [muller-etal-2026-phonological-subspace-dysarthria] Healthy speakers show tight, well-separated phonological classes along the fitted direction (d-prime = 4.2), while severe dysarthria shows the class centroids remain distinguishable but within-class distributions widen and overlap substantially (d-prime = 1.1) [muller-etal-2026-phonological-subspace-dysarthria] Measured on 890 real speakers across 10 corpora and 5 languages using a single frozen facebook/hubert-base-ls960 checkpoint, with no per-language or per-corpus fine-tuning [muller-etal-2026-phonological-subspace-dysarthria]

models: HuBERT-base · method: Difference-in-means direction

GPT-2

Outlier Dimensions Encode Task-Specific Knowledge (2023)measured

Outlier dimensions persist and can solve GLUE tasks

Details

- Outlier dimensions (variance >=5x the average) that arise in pretraining persist as outliers after fine-tuning regardless of task or seed (GPT-2's exact top-3 persist). [rudman-etal-2023-outlier-dimensions-encode-task-specific-knowledge] - A single principal outlier dimension with a learned threshold can complete GLUE binary-classification tasks, with model-dependent accuracy loss: low for GPT-2 (2.85% avg) and ALBERT (6.12%) but high for BERT (16.33%), RoBERTa (19.23%) and DistilBERT (19.12%); Pythia 70M/160M/410M lose 12.58/7.48/11.44%. [rudman-etal-2023-outlier-dimensions-encode-task-specific-knowledge] - The often-quoted "0-3.4%" figures are QNLI-only (ALBERT 3.42%, Pythia-410M 0.0%), not the cross-task averages; Pythia-410M (11.44% avg) is not low-loss. [rudman-etal-2023-outlier-dimensions-encode-task-specific-knowledge] - Tested on eight encoder and decoder models (BERT, ALBERT, DistilBERT, RoBERTa, GPT-2, Pythia-70M/160M/410M); the paper names families/sizes, not exact checkpoints. [rudman-etal-2023-outlier-dimensions-encode-task-specific-knowledge]

models: GPT-2 Small · method: Rogue-dimension analysis
The Blessing and Curse of Dimensionality in Safety Alignment (2025)measured

Linear safety separability emerges above a hidden-dimension threshold

Details

Teo, Abdullaev & Nguyen show via PCA across Qwen0.5B, GPT2-XL, Qwen7B, and Llama2-7B that emotion clusters become distinctly separable only above ~2,000-3,000 hidden dimensions [teo-etal-2025-blessing-and-curse-of-dimensionality-in-safety-alignment] This emergent linear separability is exactly what white-box jailbreaks exploit, collapsing Llama2-7B-Chat's refusal/safety from 0.97/0.99 to 0.03/0.14 [teo-etal-2025-blessing-and-curse-of-dimensionality-in-safety-alignment] Two dimension-reduction defenses (FJLT random projection and a learned bottleneck autoencoder) restore refusal/safety to 0.94-0.96, with the bottleneck retaining about 70% of benign helpfulness [teo-etal-2025-blessing-and-curse-of-dimensionality-in-safety-alignment] A Rademacher-complexity bound explains why shrinking the ambient dimension raises the sample cost of finding a steering direction [teo-etal-2025-blessing-and-curse-of-dimensionality-in-safety-alignment]

models: GPT-2 XL · method: Difference-in-means direction, Subspace-Compression Defense (FJLT / Bottleneck Autoencoder)
Large Language Models Encode Semantics and Alignment in Linearly Separable Representations (2025)measured

Topic subspaces sharpen with depth; a centroid direction induces CoT

Details

Saglam et al. study 11 autoregressive models on six arXiv topics, measuring per-layer hard-margin SVM separability and PCA intrinsic dimensionality [saglam-etal-2025-linear-subspaces] Under 10% of principal components explain nearly all variance, so topics occupy a near-affine low-dimensional subspace; separability rises toward final layers, reaching 100% SVM accuracy on all topic pairs in the largest models [saglam-etal-2025-linear-subspaces] A chain-of-thought framing distinction becomes linearly separable even more sharply than topic identity [saglam-etal-2025-linear-subspaces] Adding the CoT-vs-non-CoT centroid-difference direction to the hidden state reliably induces CoT-style responses (flagged preliminary), and a lightweight MLP guardrail built on the finding halves harmful responses [saglam-etal-2025-linear-subspaces]

models: GPT-2 XL, GPT-2 Large, GPT-2 Medium, GPT-2 Small · method: Linear probing, PCA, Activation Steering (Addition)

Custom Research ResNet (constant-width residual variant, purpose-built for SSL studies)

Reverse Engineering Self-Supervised Learning (2023)measured

SSL training drives emergent semantic-class clustering beyond augmentation invariance

Details

Ben-Shaul et al. train self-supervised models with VICReg and SimCLR objectives on a ResNet backbone, evaluated on CIFAR-100 and FOOD101 using nearest-class-mean clustering accuracy [ben-shaul-etal-2023-reverse-engineering-self-supervised-learning] The SSL objective's regularization term drives emergent clustering of representations by semantic class, not just by the sample/augmentation-invariance it explicitly optimizes [ben-shaul-etal-2023-reverse-engineering-self-supervised-learning] Semantic-class alignment increases with both training time and network depth, aligning more strongly with true classes than with equal-size random partitions [ben-shaul-etal-2023-reverse-engineering-self-supervised-learning]

models: Custom RES-L-H ResNet (trained from scratch with VICReg/SimCLR objectives on CIFAR-100/FOOD101) · method: Nearest Class-Mean (NCC) clustering accuracy, Linear probing

Saanum et al. (2024) Custom Contrastive World Models

Next State Prediction Gives Rise to Entangled, Yet Compositional Representations of Objects (2024)measured

A real contrastive world model linearly decodes which object changed despite no imposed object slots

Details

Saanum, Schulze Buschoff, Dayan & Schulz train real CNN-encoder Contrastive World Models (CWM) via next-state prediction on rendered/simulated multi-object video (Cubes, 3-body physics, Multi-dSprites, MOVi-simple, MOVi-A), taking the activation difference |encoder(x_t) minus encoder(x_t+1)| between frame pairs where exactly one object changed, and training an L1-regularized linear classifier on this difference vector to predict which object changed [saanum-etal-2024-next-state-prediction-entangled-compositional-object-representations] The linear classifier reaches near-100% held-out accuracy on Cubes and 3-body physics and about 70% (well above chance) on the harder Multi-dSprites and MOVi datasets, while untrained/random encoders and a static (non-predictive) contrastive baseline score near chance, confirming next-state prediction specifically drives the effect rather than architecture alone [saanum-etal-2024-next-state-prediction-entangled-compositional-object-representations] Despite this linear object-decodability, cosine-similarity clustering of the same difference vectors shows the distributed CWM representation is not factorized into clean per-object slots the way an explicit slot-based baseline (CSWM) is -- shared populations of units encode transformation type across different objects -- and a frozen CWM representation can still be decoded into slot-like object codes via a downstream Slot-Attention decoder, showing the compositional structure is recoverable though entangled rather than architecturally imposed [saanum-etal-2024-next-state-prediction-entangled-compositional-object-representations]

models: Contrastive World Model (Cubes / 3-body physics, from scratch), Contrastive World Model (Multi-dSprites / MOVi-simple / MOVi-A, from scratch) · method:

Dolph2Vec (Wav2Vec2-style SSL model for dolphin vocalizations)

Dolph2Vec: Self-Supervised Representations of Dolphin Vocalizations (2026)measured

A real Wav2Vec2-style model trained on longitudinal dolphin recordings organizes its codebook by whistle category and individual identity

Details

Semenzin, Mustun, Dessì, Orhan, Emanuelli, Lakretz, de Polavieja & Sumbre (2026) train Dolph2Vec, a Wav2Vec2.0-architecture model adapted for 44.1kHz audio, on ~180,000 whistles (100 hours) from 5 known bottlenose dolphins recorded over 5 real years [semenzin-etal-2026-dolph2vec-self-supervised-dolphin-vocalizations] The learned quantization codebook's specialization for signature-whistle categories is quantified via conditional entropy and mutual information (training reduces entropy 2.13->1.85 and raises MI 0.43->0.70 vs. a random-init baseline); UMAP-projected, GMM-clustered embeddings score ARI=0.3565 and NMI=0.4226 against ground-truth whistle labels, beating AVES-bio/BioLingual baselines [semenzin-etal-2026-dolph2vec-self-supervised-dolphin-vocalizations] A temporal-shuffling ablation on the feature-encoder output drops downstream classification accuracy from 82.0% to 75.1%, demonstrating dependence on genuine temporal structure [semenzin-etal-2026-dolph2vec-self-supervised-dolphin-vocalizations]

models: Dolph2Vec (Wav2Vec2.0 architecture adapted for 44.1kHz dolphin audio, trained on 100hrs/5yr real longitudinal recordings) · method: Linear probing

OpenVLA

Probing a Vision-Language-Action Model for Symbolic States and Integration into a Cognitive Architecture (2025)measured

Symbolic object-relation and action states are linearly decodable above 90% accuracy from nearly every layer of a real LIBERO-spatial-finetuned OpenVLA-7B checkpoint

Details

33 per-layer linear probes (one affine layer plus sigmoid, trained separately per layer) decode 9 binary object-relation predicates (behind, in-front-of, inside, left-of, on, on-table, right-of, open, turned-on) and 2 action-state predicates (grasped, should-move-towards) from frozen OpenVLA-7B hidden states with accuracy above 0.90 for most of the 33 layers [lu-etal-2025-probing-openvla-symbolic-states] Probed model is a real LIBERO-spatial-finetuned OpenVLA-7B checkpoint (Llama-2-7B backbone, 32 transformer blocks, 4096-dimensional hidden states per layer) [lu-etal-2025-probing-openvla-symbolic-states] The paper's own hypothesized layer-ordering (object states decoded earlier than action states) was not confirmed by the probe-accuracy pattern across layers [lu-etal-2025-probing-openvla-symbolic-states]

models: OpenVLA-7B · method: Linear probing

Custom Chess RL Agents

Reinforcement Learning in an Adaptable Chess Environment for Detecting Human-understandable Concepts (2022)measured

Concept probes show chess concepts emerge at different training points

Details

Hammersborg & Strumke train two custom self-play RL chess agents and apply TCAV-style L1-penalized logistic-regression concept probes with a random-label control for four chess concepts [hammersborg-strumke-2022-reinforcement-learning-in-an-adaptable-chess-environment-for-detecting-human-understandable-concepts] Material-advantage and mate-threat are decodable from initialization in the 6x6 ResNet agent (via skip connections), while concepts plateau after ~100 iterations in the 4x5 agent [hammersborg-strumke-2022-reinforcement-learning-in-an-adaptable-chess-environment-for-detecting-human-understandable-concepts] The work replicates McGrath et al.'s AlphaZero concept-probing methodology at a smaller toy scale [hammersborg-strumke-2022-reinforcement-learning-in-an-adaptable-chess-environment-for-detecting-human-understandable-concepts]

models: Silverman 4x5 CNN (self-play chess RL agent), Los Alamos 6x6 ResNet-CNN (self-play chess RL agent) · method: Linear probing, TCAV directional-derivative sensitivity testing

GPT-J

Large Language Models Encode Semantics and Alignment in Linearly Separable Representations (2025)measured

Topic subspaces sharpen with depth; a centroid direction induces CoT

Details

Saglam et al. study 11 autoregressive models on six arXiv topics, measuring per-layer hard-margin SVM separability and PCA intrinsic dimensionality [saglam-etal-2025-linear-subspaces] Under 10% of principal components explain nearly all variance, so topics occupy a near-affine low-dimensional subspace; separability rises toward final layers, reaching 100% SVM accuracy on all topic pairs in the largest models [saglam-etal-2025-linear-subspaces] A chain-of-thought framing distinction becomes linearly separable even more sharply than topic identity [saglam-etal-2025-linear-subspaces] Adding the CoT-vs-non-CoT centroid-difference direction to the hidden state reliably induces CoT-style responses (flagged preliminary), and a lightweight MLP guardrail built on the finding halves harmful responses [saglam-etal-2025-linear-subspaces]

models: GPT-J-6B · method: Linear probing, PCA, Activation Steering (Addition)

TAPE (Tasks Assessing Protein Embeddings)

BERTology Meets Biology: Interpreting Attention in Protein Language Models (2020)measured

Protein-LM probes recover structure; attention heads align above background

Details

Vig et al. analyze five protein language models (TapeBert, ProtBert, ProtBert-BFD, ProtAlbert, ProtXLNet) with linear single-layer probes and attention-head analysis [vig-etal-2020-bertology-meets-biology-interpreting-attention-in-protein-language-models] Individual attention heads concentrate on 3D contact maps, binding sites, and secondary structure far above background (up to 63.2% of a head's attention on contacts vs a 1.3% background; one head 49% on binding sites, Bonferroni p<0.00001) [vig-etal-2020-bertology-meets-biology-interpreting-attention-in-protein-language-models] Linear probes on output embeddings recover the same structures, quantified via precision@L/5 for contacts, precision@L/20 for binding sites, and F1 for secondary structure [vig-etal-2020-bertology-meets-biology-interpreting-attention-in-protein-language-models]

models: TapeBert (BERT-Base, Pfam-pretrained) · method: Linear probing

ProtTrans

BERTology Meets Biology: Interpreting Attention in Protein Language Models (2020)measured

Protein-LM probes recover structure; attention heads align above background

Details

Vig et al. analyze five protein language models (TapeBert, ProtBert, ProtBert-BFD, ProtAlbert, ProtXLNet) with linear single-layer probes and attention-head analysis [vig-etal-2020-bertology-meets-biology-interpreting-attention-in-protein-language-models] Individual attention heads concentrate on 3D contact maps, binding sites, and secondary structure far above background (up to 63.2% of a head's attention on contacts vs a 1.3% background; one head 49% on binding sites, Bonferroni p<0.00001) [vig-etal-2020-bertology-meets-biology-interpreting-attention-in-protein-language-models] Linear probes on output embeddings recover the same structures, quantified via precision@L/5 for contacts, precision@L/20 for binding sites, and F1 for secondary structure [vig-etal-2020-bertology-meets-biology-interpreting-attention-in-protein-language-models]

models: ProtBert, ProtBert-BFD, ProtAlbert, ProtXLNet · method: Linear probing

DAC (Descript Audio Codec)

Towards Interpretable Framework for Neural Audio Codecs via Sparse Autoencoders: A Case Study on Accent Information (2026)measured

Sparse-autoencoder decompositions of real neural audio codecs encode accent in activation magnitude or position, depending on codec type

Details

Wang, Feng, Kommineni, Lertpetchpun, Yi, Shi & Narayanan (2026) train TopK sparse autoencoders (latent ratio and sparsity swept) on mean-pooled utterance-level representations of four real neural audio codecs (EnCodec, DAC, SpeechTokenizer, Mimi), then fit logistic-regression probes on the full sparse code, on its position-only pattern, and on its magnitude-only pattern for binary accent classification (US vs. UK; US vs. non-US/UK) [wang-etal-2026-towards-interpretable-framework-for-neural-audio-codecs-via-sparse-autoencoders-accent] For DAC (acoustic-oriented), magnitude-only features preserve more accent-predictive information than position-only (DeltaF1 ~-5.0 to -5.1% vs. ~-9.8 to -12.0% at 5% sparsity); for SpeechTokenizer (phonetic-oriented, distilled from HuBERT/WavLM), the pattern reverses -- position-only preserves more than magnitude-only (~-4.5 to -6.3% vs. ~-7.1 to -7.2%) [wang-etal-2026-towards-interpretable-framework-for-neural-audio-codecs-via-sparse-autoencoders-accent] Reference (dense-representation) accent classification F1 ranges from 79.7-92.4% (US-vs-UK) across the four codecs; purely correlational, with feature steering explicitly named as future work in the paper's own conclusion [wang-etal-2026-towards-interpretable-framework-for-neural-audio-codecs-via-sparse-autoencoders-accent]

models: DAC (Descript Audio Codec) · method: TopK (k-sparse) Sparse Autoencoders, Linear probing

SpeechTokenizer

Towards Interpretable Framework for Neural Audio Codecs via Sparse Autoencoders: A Case Study on Accent Information (2026)measured

Sparse-autoencoder decompositions of real neural audio codecs encode accent in activation magnitude or position, depending on codec type

Details

Wang, Feng, Kommineni, Lertpetchpun, Yi, Shi & Narayanan (2026) train TopK sparse autoencoders (latent ratio and sparsity swept) on mean-pooled utterance-level representations of four real neural audio codecs (EnCodec, DAC, SpeechTokenizer, Mimi), then fit logistic-regression probes on the full sparse code, on its position-only pattern, and on its magnitude-only pattern for binary accent classification (US vs. UK; US vs. non-US/UK) [wang-etal-2026-towards-interpretable-framework-for-neural-audio-codecs-via-sparse-autoencoders-accent] For DAC (acoustic-oriented), magnitude-only features preserve more accent-predictive information than position-only (DeltaF1 ~-5.0 to -5.1% vs. ~-9.8 to -12.0% at 5% sparsity); for SpeechTokenizer (phonetic-oriented, distilled from HuBERT/WavLM), the pattern reverses -- position-only preserves more than magnitude-only (~-4.5 to -6.3% vs. ~-7.1 to -7.2%) [wang-etal-2026-towards-interpretable-framework-for-neural-audio-codecs-via-sparse-autoencoders-accent] Reference (dense-representation) accent classification F1 ranges from 79.7-92.4% (US-vs-UK) across the four codecs; purely correlational, with feature steering explicitly named as future work in the paper's own conclusion [wang-etal-2026-towards-interpretable-framework-for-neural-audio-codecs-via-sparse-autoencoders-accent]

models: SpeechTokenizer · method: TopK (k-sparse) Sparse Autoencoders, Linear probing

Jukebox

Do Music Generation Models Encode Music Theory? (SynTheory) (2024)measured

MLP probes decode seven music-theory concepts from generative models

Details

Wei et al. introduce a synthetic dataset isolating seven music-theory concepts and train two-layer MLP probes on internal representations of Jukebox and MusicGen small/medium/large [wei-freeman-etal-2024-do-music-generation-models-encode-music-theory-syntheory] Probes on the language-model stage reach mean decodability up to 0.984 (Jukebox), 0.950 (MusicGen-Small), 0.914 (Medium), and 0.929 (Large) [wei-freeman-etal-2024-do-music-generation-models-encode-music-theory-syntheory] The MusicGen audio-codec stage alone reaches only 0.701, so decodability depends on which processing stage and model scale are probed [wei-freeman-etal-2024-do-music-generation-models-encode-music-theory-syntheory]

models: Jukebox · method: Mlp probing

MusicGen

Do Music Generation Models Encode Music Theory? (SynTheory) (2024)measured

MLP probes decode seven music-theory concepts from generative models

Details

Wei et al. introduce a synthetic dataset isolating seven music-theory concepts and train two-layer MLP probes on internal representations of Jukebox and MusicGen small/medium/large [wei-freeman-etal-2024-do-music-generation-models-encode-music-theory-syntheory] Probes on the language-model stage reach mean decodability up to 0.984 (Jukebox), 0.950 (MusicGen-Small), 0.914 (Medium), and 0.929 (Large) [wei-freeman-etal-2024-do-music-generation-models-encode-music-theory-syntheory] The MusicGen audio-codec stage alone reaches only 0.701, so decodability depends on which processing stage and model scale are probed [wei-freeman-etal-2024-do-music-generation-models-encode-music-theory-syntheory]

models: MusicGen-Small, MusicGen-Large · method: Mlp probing

Falcon3

Monitoring Emergent Reward Hacking During Generation via Internal Activations (2026)measured

A probe on SAE-PCA activations detects reward hacking token-by-token

Details

Wilhelm, Wittkopp & Kao train a per-layer sparse autoencoder on the last four layers of Qwen2.5-Instruct-7B, LLaMA-3.1-8B, and Falcon3-7B, then fit a logistic-regression probe on PCA-reduced SAE features for a per-token hack probability [wilhelm-wittkopp-kao-2026-monitoring-emergent-reward-hacking-during-generation-via-internal-activations] Against a GPT-4o judge (metric F1), the probe reaches F1=1.000 only on the benign control-adapter data, while reward-hacking detection F1 is 0.760-0.961 (e.g. Llama 0.961, Qwen 0.784) [wilhelm-wittkopp-kao-2026-monitoring-emergent-reward-hacking-during-generation-via-internal-activations] Reward-hacking versus benign activations are linearly separable in the SAE+PCA-reduced space; no causal steering is performed [wilhelm-wittkopp-kao-2026-monitoring-emergent-reward-hacking-during-generation-via-internal-activations]

models: Falcon3-7B · method: Sparse Autoencoders (SAE), PCA, Linear probing