MATH · IN · MODELS

OPT

Meta AI (Facebook AI Research)

Structures found in this family (5)

Cone, Anisotropy, Intrinsic-dimension profile across depth, Linear Subspace, Linear Direction

By model (6)

OPT-1.3B · 1.3B
Cone, Intrinsic-dimension profile across depth, Linear Direction
OPT-6.7B · 6.7B
Cone, Linear Subspace, Linear Direction
OPT-125M · 125M
Intrinsic-dimension profile across depth, Linear Direction
OPT-13B · 13B
Intrinsic-dimension profile across depth, Anisotropy
OPT-2.7B · 2.7B
Cone, Anisotropy
OPT-350M · 350M
Anisotropy

Observations (6)

A cone probe isolates a causal tree-like subspace for genealogy
RoPE massive values concentrate at fixed Q/K dimensions
Layer intrinsic-dimension peak aligns with the best brain-predicting layer
A projected-out toxic subspace beats DPO more sample-efficiently
Decoder-only models peak anisotropy in middle layers
The Tuned Lens replaces the logit lens with a trained per-layer affine correction

Papers

Investigating Representation Universality: Case Study on Genealogical Representations (2024), Massive Values in Self-Attention Modules are the Key to Contextual Knowledge Understanding (2025), Abstraction Induces the Brain Alignment of Language and Speech Models (2026), Model Editing as a Robust and Denoised Variant of DPO: A Case Study on Toxicity (2024), The Shape of Learning: Anisotropy and Intrinsic Dimensions in Transformer-Based Models (2024), Eliciting Latent Predictions from Transformers with the Tuned Lens (2023)