A linear map from 10K pairs aligns English BERT with multilingual BERT

Wang et al. fit a linear map from monolingual English BERT to multilingual BERT using contextual word alignments from only 10,000 Europarl sentence pairs per language [wang-etal-2019-clbt] They compare closed-form orthogonal Procrustes against an unconstrained gradient-descent linear map; the unconstrained map is marginally better and a nonlinear map adds nothing [wang-etal-2019-clbt] On zero-shot cross-lingual dependency parsing (17 languages), CLBT beats the static-embedding SOTA (avg LAS 60.63->63.54) and matches XLM despite using 10K versus millions of parallel sentences [wang-etal-2019-clbt] The alignment claim rests on downstream transfer performance rather than an independent similarity statistic (only a qualitative t-SNE case study is shown) [wang-etal-2019-clbt]