Independent monolingual LMs learn alignable isomorphic geometries

measured in 1 paper

Conneau et al. (2020) train five same-architecture masked LMs from scratch, one per language (en/fr/de/ru/zh), with no shared parameters, vocabulary or parallel data [conneau-etal-2020] Orthogonal Procrustes mapping between language pairs recovers substantial cross-lingual structure at subword, word and sentence granularity, improving at higher layers [conneau-etal-2020] CKA between paired parallel-sentence representations shows monolingual models are more similar to each other than to a random encoder but less than bilingually co-trained models (e.g. en-fr 0.58/0.59 vs 0.69 vs 0.46) [conneau-etal-2020] CKA similarity correlates with alignment-based sentence-retrieval performance at >0.9 Pearson, evidencing universal latent symmetries across independently-trained models [conneau-etal-2020]

Structure

Platonic Representation Hypothesis, Linear Subspace

Context

cross-lingual transfer, representational isomorphism, orthogonal Procrustes, centered kernel alignment, universal latent symmetries

Confirmed in models

Conneau et al. Monolingual MLM (English) · Conneau et al. Monolingual MLM, Conneau et al. Monolingual MLM (French) · Conneau et al. Monolingual MLM, Conneau et al. Monolingual MLM (German) · Conneau et al. Monolingual MLM, Conneau et al. Monolingual MLM (Russian) · Conneau et al. Monolingual MLM, Conneau et al. Monolingual MLM (Chinese) · Conneau et al. Monolingual MLM

Method

Orthogonal Procrustes alignment, Centered Kernel Alignment (CKA)

Papers

Emerging Cross-lingual Structure in Pretrained Language Models — Conneau, Alexis, Wu, Shijie, Li, Haoran, Zettlemoyer, Luke, Stoyanov, Veselin2020 · arXiv:1911.01464