A fitted linear metric embeds dependency trees in BERT/ELMo geometry

Hewitt & Manning introduce the structural probe: a linear transform B under which squared Euclidean distance approximates dependency-tree edge distance and squared norm approximates depth from the parse root [hewitt-manning-2019] On ELMo and BERT (base-cased, large-cased) the probe recovers trees far better than non-contextual baselines (BERT-large layer 16: 81.7% UUAS, 0.87 distance-Spearman) [hewitt-manning-2019] The effective rank plateaus around 64-128 dimensions, well below ambient hidden size, so the tree occupies a compact subspace [hewitt-manning-2019] Chen et al. fit the same distance/depth objectives in a Poincare ball and recover tree structure consistently better than the Euclidean probe on contextualized representations (BERT-base layer 7 UUAS 79.8% to 83.7%), with no gain on non-contextual baselines [chen-etal-2021] They explicitly caution this shows a hyperbolic probe fits better, not that BERT's syntactic subspace is itself intrinsically hyperbolic [chen-etal-2021]