Whitening matches BERT-flow at far lower dimension

measured in 1 paper

- Whitening (mean-center plus PCA whitening) makes BERT embeddings isotropic and ranks dimensions by variance, so the whitening map can be truncated to k dimensions with little loss. [su-etal-2021] - Truncated to just k=109 dimensions, BERT-base-whitening (NLI) reaches SICK-R Spearman 66.52, beating full 768-dimensional BERT-flow (NLI) at 65.44 by +1.08. [su-etal-2021] - The advantage over BERT-flow is on SICK-R (not STS-B), and it is achieved at far lower dimensionality. [su-etal-2021] - Tested on BERT-base and BERT-large (uncased). [su-etal-2021]

Structure

Anisotropy

Context

orthonormal basis, dimensionality reduction, incremental covariance estimation, retrieval efficiency

Confirmed in models

BERT-base-uncased · BERT, BERT-large-uncased · BERT

Method

Whitening transformation

Papers

Whitening Sentence Representations for Better Semantics and Faster Retrieval — Su, Jianlin, Cao, Jiarun, Liu, Weijie, Ou, Yangyiwen2021 · arXiv:2103.15316