Clusters and low-dim manifolds explain apparent anisotropy

- Apparent global anisotropy (inter-type cosine ~0.7) is largely an artifact of a few isolated clusters: after k-means clustering and per-cluster mean-shifting, within-cluster cosine isotropy is near zero across every layer. [cai-etal-2021] - Contextual embeddings sit on low intrinsic-dimension manifolds, with Local Intrinsic Dimension 5.6 (BERT), 7.3 (DistilBERT), 6.8 (GPT), 7.0 (GPT-2), 9.1 (ELMo), far below the 768/1024 ambient dimensions. [cai-etal-2021] - GPT and GPT-2 additionally show a Swiss-Roll manifold tied to word frequency. [cai-etal-2021] - Tested on uncased base BERT, DistilBERT, GPT and GPT-2 (Hugging Face) and AllenNLP ELMo (no 5.5B variant); observational. [cai-etal-2021]