Word2vec-like models learn orthogonal linear subspaces one at a time, each incrementing the effective rank of the embeddings until capacity saturates, and the top subspaces are interpretable topic-level concepts when trained on Wikipedia

- Karkada, Simon, Bahri and DeWeese analytically solve the gradient-flow dynamics of a quartic approximation of the word2vec loss, showing that the resulting dynamics and downstream performance match word2vec empirically, that the embeddings learn orthogonal linear subspaces one at a time (each incrementing effective rank until capacity saturates), that each top subspace represents an interpretable topic-level concept when trained on Wikipedia, and that linear representations of more abstract concepts emerge during training and complete analogies by vector addition. Theory-first: the closed-form solutions are for the authors' quadratic word embedding models (QWEMs); word2vec is the empirical comparison (Fig. 1: stepwise subspace growth in both), so directional wording. [karkada-etal-2025-closed-form-training-dynamics-reveal-learned-features-and-linear-struc]