Word embeddings share a mean plus top-PC anisotropy
measured in 1 paper- Word embeddings share a large nonzero common mean vector plus energy concentrated in ~8-10 dominant directions that encode frequency, making them strongly anisotropic. [mu-viswanath-2018-all-but-the-top] - The all-but-the-top postprocessing (subtract the mean, project out the top D principal components; D=3 for word2vec, D=2 for GloVe, rule D~=d/100) makes vectors more isotropic. [mu-viswanath-2018-all-but-the-top] - The correction causally improves downstream tasks with no retraining: word similarity +1.7% average (GloVe Rare-Words +5.63%), STS +4%, and text classification in 34 of 40 settings (+2.85%). [mu-viswanath-2018-all-but-the-top] - Measured on word2vec (GoogleNews) and GloVe (Common Crawl, 2.2M-vocab / 840B). [mu-viswanath-2018-all-but-the-top]