scGPT gene-embedding effective rank collapses 14-fold across layers

measured in 1 paper

Kendiukhov performs per-layer SVD on scGPT's gene-embedding matrix across its 12 layers; effective rank collapses monotonically 23.6 to 1.6 (Spearman rho=-1.000) and the top singular vector's variance fraction rises 53.7% to 93.4% [kendiukhov-2026-scgpt-spectral-geometry] TwoNN intrinsic dimension falls 32.6 to 18.1 and the participation ratio drops 6.1-fold, while a feature-shuffle control rebounds effective rank to 28.9, confirming a genuine (non-artifactual) collapse [kendiukhov-2026-scgpt-spectral-geometry] The final compressed layer's low-rank subspace matches independent biology: STRING PPI co-pole rate 0.226 vs 0.124 null, TRRUST TF-target AUROC up to 0.789, and cell-type marker AUROC 0.851 vs 0.488 chance [kendiukhov-2026-scgpt-spectral-geometry] Extensive confound controls reject persistent-homology and feed-forward-loop explanations, and no causal intervention is performed [kendiukhov-2026-scgpt-spectral-geometry]

Structure

Intrinsic-dimension profile across depth

Context

SVD effective-rank collapse across scGPT's 12 layers (23.6 to 1.6, rho=-1.000), top-singular-vector variance concentration (53.7% to 93.4%), TwoNN intrinsic dimensionality and participation-ratio cross-checks, feature-shuffle control confirming genuine (non-artifactual) collapse, final-layer low-rank subspace correlates with PPI/TF-target/cell-type ground truth

Confirmed in models

scGPT (whole-human pretrained checkpoint) · scGPT

Method

Intrinsic dimension estimation (TwoNN)

Papers

Multi-Dimensional Spectral Geometry of Biological Knowledge in Single-Cell Transformer Representations — Kendiukhov, Ihor2026