IsoScore shows contextual spaces use even fewer dimensions
measured in 1 paper- IsoScore measures isotropy from the covariance alone (PCA-reorient, length-normalize the variance vector, map to [0,1]), making it mean-agnostic, scale- and rotation-invariant. [rudman-etal-2021] - Every contextualized model scores below 0.18 (GPT and GPT-2 do not isotropically use even one dimension), so they use fewer dimensions than cosine-based estimates implied. [rudman-etal-2021] - IsoScore overturns two prior conclusions: isotropy does not increase in earlier layers, and mean-centering does not recover isotropy (a rotation test holds IsoScore at 0.216 while average cosine swings 0.97-0.99). [rudman-etal-2021] - Computed on WikiText-2 token embeddings from GPT (110M), GPT-2 (117M), BERT-base-uncased and DistilBERT-base-uncased. [rudman-etal-2021]