A Backpack model sense-vector topology partially transfers across languages

measured in 1 paper

Cruz, Adelani & Aji adapt a Backpack LM (GPT-2-based, K learned sense vectors per word) from English to Estonian, Turkish, Indonesian and Swahili [cruz-etal-2026-multilinguality-as-sense-adaptation] Sense-topology correlation between English and target-language sense-Gram matrices reaches rho ~0.25-0.30 for SENSIA versus ~0.16-0.21 control, a consistent ~0.09 gap [cruz-etal-2026-multilinguality-as-sense-adaptation] Orthogonal Procrustes alignment to English gives post-alignment cosine 0.35-0.44 versus 0.26-0.33 control [cruz-etal-2026-multilinguality-as-sense-adaptation] Collapsing the soft sense-mixture to top-1 raises FLORES cross-entropy by roughly +6 across all four languages, tying the sense-mixture mechanism to translation quality [cruz-etal-2026-multilinguality-as-sense-adaptation]

Structure

Linear Subspace

Context

cross-lingual sense-vector manifold partially preserves source-language cosine topology, orthogonal Procrustes alignment of sense manifolds across languages, above a matched control, ablating the soft sense-mixture weights degrades cross-lingual translation quality

Confirmed in models

GPT-2-small · GPT, GPT-2-Medium · GPT, GPT-2-Large · GPT

Method

Representational Similarity Analysis (RSA), Orthogonal Procrustes alignment, Causal interventions (steering)

Papers

Multilinguality as Sense Adaptation — Cruz, Jan Christian Blaise, Adelani, David Ifeoluwa, Aji, Alham Fikri2026 · arXiv:2601.10310