ELMo

AllenNLP / Allen Institute for AI

Structures found in this family (5)

Anisotropy, Linear Separability, Intrinsic-dimension profile across depth, Linear Subspace, Tree Metric Embedding

By model (2)

ELMo (5.5B-word pretrained, 2-layer biLSTM)

Linear Separability, Linear Subspace, Tree Metric Embedding

ELMo (AllenNLP biLM, 1B Word Benchmark) · 94M

Anisotropy, Intrinsic-dimension profile across depth

Observations (6)

Contextualized embeddings occupy a narrow anisotropic cone

Clusters and low-dim manifolds explain apparent anisotropy

DirectProbe finds convex label clusters whose geometry predicts accuracy

Contextualized embeddings lie on a low-dimensional manifold rising with depth

Linguistic categories occupy nested low-dimensional subspaces

A fitted linear metric embeds dependency trees in BERT/ELMo geometry

Papers

How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings (2019), Isotropy in the Contextual Embedding Space: Clusters and Manifolds (2021), DirectProbe: Studying Representations without Classifiers (2021), The Low-Dimensional Linear Geometry of Contextualized Word Representations (2021), A Structural Probe for Finding Syntax in Word Representations (2019), Probing BERT in Hyperbolic Spaces (2021)