Normalizing flow removes BERT's anisotropic frequency holes
measured in 1 paper- BERT's sentence-embedding space is anisotropic and frequency-biased: high-frequency words sit near the origin (norm ~0.95) and low-frequency words far out (~1.45), leaving sparse low-density "holes". [li-etal-2020-bertflow] - A Glow-style invertible normalizing flow maps the frozen embeddings to an isotropic standard Gaussian, correcting anisotropy without discarding directions. [li-etal-2020-bertflow] - Flow calibration raises average STS Spearman from 62.89 to 70.98 (BERT-base) and 66.57 to 74.73 (BERT-large) without NLI supervision. [li-etal-2020-bertflow] - Tested on BERT-base and BERT-large (paper writes "BERT_base"/"BERT_large"; uncased per released code); SBERT numbers are cited baselines only. [li-etal-2020-bertflow]
Structure
Context
word frequency bias, embedding space holes, normalizing flows, lexical vs. semantic similarity
Confirmed in models
Papers
On the Sentence Embeddings from Pre-trained Language Models — Li, Bohan, Zhou, Hao, He, Junxian, Wang, Mingxuan, Yang, Yiming, Li, Lei