Sparse dimension identification (axis-aligned language-specific dimensions)

methods / Direction Extraction / Sparse dimension identification (axis-aligned language-specific dimensions)

Sparse dimension identification (axis-aligned language-specific dimensions)

Techniquebeginner

Identifies a small, sparse set of ambient-basis (axis-aligned) dimensions responsible for a cross-condition transition — e.g. language-agnostic to language-specific representation — by comparing corpus-mean activations across two conditions and keeping the top-K dimensions with the largest absolute difference, then validating by directly overwriting only those dimensions at inference time.

Used in (1 observation)

A sparse axis-aligned dimension set causally controls output languagemeasured

structure: Linear Subspace · models: Llama-2-7B, Llama-2-13B, Llama-3.1-8B, Aya-23-8B · paper: Language Lives in Sparse Dimensions: Toward Interpretable and Efficient Multilingual Control for Large Language Models