Protein-LM shape spaces expand then compress at low absolute dimension

Beshkov & Malthe-Sorenssen treat each protein's per-residue PLM representation as a curve, map it to a square-root-velocity shape space to quotient out rotation and translation, and measure Frechet radius and tangent-PCA effective dimension layer-by-layer across ESM2 (35M/150M/650M/3B) and Ankh on 1,377 SCOPe proteins [beshkov-malthe-sorenssen-2026-towards-understanding-the-shape-of-representations-in-protein-language-models] Frechet radius decreases with depth and is far smaller for PLM shape space than for real 3D protein structure, largely independent of model size [beshkov-malthe-sorenssen-2026-towards-understanding-the-shape-of-representations-in-protein-language-models] Tangent-PCA effective dimension shows an expansion-then-compression trajectory across layers, more pronounced for larger models, at far lower absolute dimension than PCA on flattened pointwise activations [beshkov-malthe-sorenssen-2026-towards-understanding-the-shape-of-representations-in-protein-language-models] A graph-filtration comparison finds 3D structure is most faithfully encoded at short (2-residue) and moderate (8-residue) context, peaking just before the final layer in every model [beshkov-malthe-sorenssen-2026-towards-understanding-the-shape-of-representations-in-protein-language-models]