A dominant unsupervised PCA axis in real ESM protein language models orders sequences along a cellular-to-viral nativeness continuum

Bigot, Bhasin, Park, Shakhnovich & Wang (2026) compute the first principal component of pooled per-residue embeddings from real ESM2 (8M-15B), ESMC, and ESM3 checkpoints over cellular, viral, and shuffled-control sequences, finding PC1 correlates strongly with masked-reconstruction perplexity (Spearman rho up to 0.961, explaining 54-73% of variance) [bigot-etal-2026-viral-proteins-reveal-geometry-of-protein-language-models] An L2-regularized logistic-regression probe on the same embeddings reaches AUC-ROC 0.97-1.00 for larger models -- an unsupervised dominant linear axis plus linear-probe confirmation, purely observational (no causal validation) [bigot-etal-2026-viral-proteins-reveal-geometry-of-protein-language-models]