Multilingual NMT encoders converge across languages with depth

Kudugunta et al. train one shared 375M Transformer NMT model on over 25B pairs across 103 languages, then compare per-language representations with SVCCA [kudugunta-etal-2019] For X->En, encoder representations of different source languages become MORE similar moving up the encoder; for En->X, the same English input's encoder representations DIVERGE by target language [kudugunta-etal-2019] Embedding-layer representations are highly stable under per-language-pair fine-tuning (mean canonical correlation >0.98), with later layers more sensitive [kudugunta-etal-2019] Encoder representations visually cluster by language family in Laplacian-eigenmap embeddings, but this is qualitative with no clustering statistic; no activation intervention is performed [kudugunta-etal-2019]