MATH · IN · MODELS
methods / Representation Alignment / Representational Similarity Analysis (RSA) / Subspace PGA (Predictive Geometric Alignment)

Subspace PGA (Predictive Geometric Alignment)

Techniqueadvanced

Tests whether a layer's full representation-space distance structure is significantly better preserved by projecting onto the unembedding matrix's top-k readout subspace than by projecting onto a random subspace of the same dimension — a z-scored RSA variant that asks specifically whether intermediate-layer geometry is organized around the subspace the model will eventually read predictions out of.

Used in (1 observation)

structure: Linear Subspace · models: Pythia-70M, Pythia-160M, Pythia-410M, Pythia-1B, Pythia-1.4B, Pythia-2.8B, Pythia-6.9B, OLMo-1B, Phi-1.5, Gemma-2-2B · paper: Scale Determines Whether Language Models Organize Representation Geometry for Prediction