MultiBERTs
Google Research
Structures found in this family (1)
By model (5)
MultiBERTs (seed 0, tracked across 2M pretraining steps) · 110M (BERT-base architecture)
MultiBERTs (seed 1, tracked across pretraining steps) · 110M (BERT-base architecture)
MultiBERTs (seed 2, tracked across pretraining steps) · 110M (BERT-base architecture)
MultiBERTs (seed 3, tracked across pretraining steps) · 110M (BERT-base architecture)
MultiBERTs (seed 4, tracked across pretraining steps) · 110M (BERT-base architecture)
Papers
Subspace Chronicles: How Linguistic Information Emerges, Shifts and Interacts during Language Model Training (2023)