A Hewitt-Manning-style distance probe recovers AST tree distances from CodeBERT and GraphCodeBERT, peaking at layer 5

Wan et al. fit a linear map B so that squared Euclidean distance between token vectors approximates the number of AST edges between the tokens, on CodeBERT and GraphCodeBERT (12 layers, 768 dimensions) [wan-etal-2022-what-do-they-capture] On Python the average distance Spearman correlation is 0.85 for CodeBERT layer 5 and 0.86 for GraphCodeBERT layer 5, against 0.60 for the pre-Transformer embedding layer (CodeBERT-0) (Table 2) [wan-etal-2022-what-do-they-capture] The best layer is layer 5 for both models (the text reports 84% and 86%), and GraphCodeBERT is above CodeBERT in every layer across Python, Java and PHP [wan-etal-2022-what-do-they-capture] Probe-only evidence: no rank sweep and no causal intervention are performed; the dimensionality estimate of this structure is in the AST-Probe record [wan-etal-2022-what-do-they-capture]