A sparse axis-aligned dimension set causally controls output language

measured in 1 paper

Zhong et al. hypothesize the English-centric cross-lingual transition is governed by a small, layer-consistent set of dimensions, identified from as few as ~50 sentences by comparing corpus-mean activations [zhong-etal-2025-language-lives-in-sparse-dimensions] Keeping the top-400 dimensions (~8-10% of hidden size), cross-language overlap tracks typological similarity (Chinese/Japanese share 193/400) and the monolingual and parallel methods agree 77.6% [zhong-etal-2025-language-lives-in-sparse-dimensions] Overwriting only those dimensions at one intermediate layer with a scaled target-language mean switches the output language while preserving semantic content (BLEU) across Llama-2/3.1 and Aya23-8B [zhong-etal-2025-language-lives-in-sparse-dimensions] It outperforms neuron-level baselines by up to 12.69 points at much lower data/compute cost, holding across most intermediate layers [zhong-etal-2025-language-lives-in-sparse-dimensions]

Structure

Linear Subspace

Context

sparse, layer-consistent axis-aligned dimensions governing cross-lingual transition, training-free identification from as few as 50 sentences (monolingual or parallel), cross-language overlap in identified dimensions tracking typological similarity, dimension-overwrite intervention switching output language while preserving semantic content, outperforming neuron-level baselines at substantially lower data/compute cost

Confirmed in models

Llama-2-7B · Llama, Llama-2-13B · Llama, Llama-3.1-8B · Llama, Aya-23-8B · Aya

Method

Sparse dimension identification (axis-aligned language-specific dimensions)

Papers

Language Lives in Sparse Dimensions: Toward Interpretable and Efficient Multilingual Control for Large Language Models — Zhong, Chengzhi, Cheng, Fei, Liu, Qianying, Murawaki, Yugo, Chu, Chenhui, Kurohashi, Sadao2025 · arXiv:2510.07213