Conneau et al. Monolingual MLM
Facebook AI / Johns Hopkins (Conneau, Wu, Li, Zettlemoyer & Stoyanov 2020; custom 8-layer bidirectional Transformer MLM, d_model=768, 12 heads, FFN width 3072, GELU, trained from scratch per-language on Wikipedia using the Lample & Conneau (2019) MLM implementation, architecture held fixed across languages for controlled comparison)
Structures found in this family (2)
By model (5)
Conneau et al. Monolingual MLM (German)
Conneau et al. Monolingual MLM (English)
Conneau et al. Monolingual MLM (French)
Conneau et al. Monolingual MLM (Russian)
Conneau et al. Monolingual MLM (Chinese)
Papers
Emerging Cross-lingual Structure in Pretrained Language Models (2020)