Syntactic relations are encoded as directions, not just distances
measured in 1 paper- A supervised "Polar Probe" on frozen LLM activations encodes each dependency edge by both distance and relative direction, making same-type relation embeddings collinear and different-type ones orthogonal. [diego-simon-etal-2024] - This extends Hewitt-Manning's distance-only tree metric with an angular component, so a dependency tree's relations become readable as directions in activation space. [diego-simon-etal-2024] - The Polar Probe reaches AUC 95%, versus 74% for a distance-only Structural Probe, 80% for no probe, and 50% on a randomly initialized model (optimal probe dimensionality 128). [diego-simon-etal-2024] - Tested on Llama-2-7B, Mistral-7B-v0.1 and BERT-large (casing unspecified); the collinearity is imposed by the probe's supervised objective rather than discovered, and there is no causal intervention. [diego-simon-etal-2024]