Contextuality metrics (self-similarity, intra-sentence similarity, MEV)

methods / Theoretical / Analytical / Geometric analysis / Contextuality metrics (self-similarity, intra-sentence similarity, MEV)

Contextuality metrics (self-similarity, intra-sentence similarity, MEV)

Techniqueintermediate

Three anisotropy-adjusted statistics — self-similarity, intra-sentence similarity, and maximum explainable variance (MEV) — that quantify how context-specific a contextualized word representation is, after subtracting out each layer's own ambient anisotropy baseline so the numbers aren't confounded by directional bunching alone.

Used in (1 observation)

Contextualized embeddings occupy a narrow anisotropic conemeasured

structure: Anisotropy · models: ELMo (AllenNLP biLM, 1B Word Benchmark), BERT-base-cased, GPT-2-small · paper: How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings