ProGen2

Salesforce Research

Structures found in this family (3)

Linear Direction, Linear Separability, Anisotropy

By model (1)

ProGen2-base · 764M

Linear Direction, Linear Separability, Anisotropy

Observations (2)

A protein-LM nativeness axis orders sequences; viral signal stays separable

Dimension-level activation outliers, measured as the ratio gamma of mean norm to per-token variation, predict sparse-autoencoder feature death across 454 model-layer combinations spanning language, vision, protein and genomic models

Papers

Viral Proteins Reveal Geometry of Protein Language Models (2026), On the Relationship Between Activation Outliers and Feature Death in Sparse Autoencoders (2026)