MATH · IN · MODELS

ESM-2

Meta AI (FAIR)

Structures found in this family (4)

Intrinsic-dimension profile across depth, Linear Direction, Linear Separability, Anisotropy

Hypotheses argued for by this family (1)

Spatial-Functional Modularity Hypothesis

By model (5)

ESM-2 (3B) · 3B
Intrinsic-dimension profile across depth, Anisotropy, Linear Separability, Linear Direction
ESM-2 (650M) · 650M
Intrinsic-dimension profile across depth, Linear Direction, Linear Separability, Anisotropy
ESM-2 (35M) · 35M
Intrinsic-dimension profile across depth, Anisotropy, Linear Separability
ESM-2 (150M) · 150M
Intrinsic-dimension profile across depth, Linear Separability
ESM-2 (15B) · 15B
Linear Direction, Linear Separability

Observations (8)

Protein-LM shape spaces expand then compress at low absolute dimension
A dominant unsupervised PCA axis in real ESM protein language models orders sequences along a cellular-to-viral nativeness continuum
A protein-LM nativeness axis orders sequences; viral signal stays separable
Dimension-level activation outliers, measured as the ratio gamma of mean norm to per-token variation, predict sparse-autoencoder feature death across 454 model-layer combinations spanning language, vision, protein and genomic models
ESM attention maps are linearly readable as contact maps in one pass
Intrinsic dimension expands then compresses; semantics peak at the minimum
Steering an ESM2-3B hydrophobicity SAE direction shifts predicted solvent accessibility
TopK sparse-autoencoder features on a real fine-tuned ESM2 align with real protein structural sites, and steering them causally improves in-silico protein design

Papers

Towards Understanding the Shape of Representations in Protein Language Models (2026), Viral Proteins Reveal Geometry of Protein Language Models (2026), On the Relationship Between Activation Outliers and Feature Death in Sparse Autoencoders (2026), Protein Contacts Are Already in the Attention: A Single-Forward-Pass Alternative to the Categorical Jacobian (2026), The Geometry of Hidden Representations of Large Transformer Models (2023), The Geometry of Concepts: Sparse Autoencoder Feature Structure (2024), Towards Interpretable Protein Structure Prediction with Sparse Autoencoders (2025), Sparse Autoencoders for Low-N Protein Function Prediction and Design (2025)