Neighborhood overlap

methods / Theoretical / Analytical / Neighborhood overlap

Techniquebeginner

Measures how much a point's set of k-nearest-neighbors changes between two representations — between successive layers (tracking how fast a layer rewrites the data's local structure) or between a representation and ground-truth class labels (tracking how much semantic information the representation carries).

Used in (1 observation)

Intrinsic dimension expands then compresses; semantics peak at the minimummeasured

structure: Intrinsic-dimension profile across depth · models: ESM-2 (35M), ESM-2 (650M), ESM-2 (3B), iGPT-S, iGPT-M, iGPT-L, Llama-2-70B, GPT-2-XL, Gemma-2-2B · paper: The Geometry of Hidden Representations of Large Transformer Models, The Geometry of Concepts: Sparse Autoencoder Feature Structure