MATH · IN · MODELS
methods / Direction Extraction / Minimal subspace search (rank-sweep probing)

Minimal subspace search (rank-sweep probing)

Techniqueintermediate

Sweeps candidate subspace dimension d, jointly training a rank-d linear projection plus classifier at each d, to find the smallest d at which held-out accuracy is within a fixed tolerance of the unconstrained ceiling — operationalizes 'how many dimensions does this concept need', distinct from simply asking whether it is decodable at all.

Used in (1 observation)

structure: Linear Subspace · models: ELMo (5.5B-word pretrained, 2-layer biLSTM), BERT-base-cased · paper: The Low-Dimensional Linear Geometry of Contextualized Word Representations