Subspace-similarity probing

methods / Theoretical / Analytical / Subspace-similarity probing

Techniqueintermediate

Fits a linear probe per task and compares the resulting probe subspaces directly to each other (rather than only reporting each probe's own accuracy), letting a single procedure track how much information distinct tasks' subspaces share, and how that sharing changes across training checkpoints.

Used in (1 observation)

Linguistic subspaces emerge then specialize in critical pretraining phasesmeasured

structure: Linear Subspace · models: MultiBERTs (seed 0, tracked across 2M pretraining steps), MultiBERTs (seed 1, tracked across pretraining steps), MultiBERTs (seed 2, tracked across pretraining steps), MultiBERTs (seed 3, tracked across pretraining steps), MultiBERTs (seed 4, tracked across pretraining steps) · paper: Subspace Chronicles: How Linguistic Information Emerges, Shifts and Interacts during Language Model Training