DistilBERT

Hugging Face

Structures found in this family (3)

Anisotropy, Linear Separability, Intrinsic-dimension profile across depth

By model (1)

DistilBERT-base-uncased · 66M

Anisotropy, Linear Separability, Intrinsic-dimension profile across depth

Observations (6)

Clusters and low-dim manifolds explain apparent anisotropy

Linguistic category manifolds untangle across transformer layers

IsoScore shows contextual spaces use even fewer dimensions

Contextualized embeddings lie on a low-dimensional manifold rising with depth

Outlier dimensions persist and can solve GLUE tasks

Whitening removes anisotropy and improves sentence similarity

Papers

Isotropy in the Contextual Embedding Space: Clusters and Manifolds (2021), Emergence of Separable Manifolds in Deep Language Representations (2020), IsoScore: Measuring the Uniformity of Embedding Space Utilization (2021), Outlier Dimensions Encode Task-Specific Knowledge (2023), WhiteningBERT: An Easy Unsupervised Sentence Embedding Approach (2021)