AST-Probe recovers full code syntax trees from a 64-128 dimensional subspace

measured in 1 paper

Hernandez Lopez et al. extend Hewitt-Manning's structural probe to programming-language abstract syntax trees, fitting an orthogonal subspace projection predicting a (distance, label, marker) tuple convertible to the complete labeled AST [hernandez-lopez-etal-2022-ast-probe] Across five pretrained code/text models plus a random-init control, all five show a significant F1 gap over the baseline across Python, JavaScript, and Go (GraphCodeBERT and CodeBERT best) [hernandez-lopez-etal-2022-ast-probe] The syntactic subspace's dimensionality is 64-128 of 768 ambient dimensions (8-17%), concentrated in middle layers [hernandez-lopez-etal-2022-ast-probe] No causal intervention is performed; the addition rests on the quantified subspace-dimensionality claim [hernandez-lopez-etal-2022-ast-probe]

Structure

Tree Metric Embedding

Context

AST-Probe: orthogonal-subspace projection predicting (distance, label, marker) tuple, full AST recoverability (not just pairwise distance/depth) across 5 code/text models, syntactic-subspace dimensionality estimate (64-128 of 768 ambient dimensions), middle-layer concentration of AST information, randomly-initialized baseline control confirming the effect is learned, not architectural

Confirmed in models

CodeBERT-base · CodeBERT, GraphCodeBERT-base · GraphCodeBERT, CodeT5-base · CodeT5, RoBERTa-base · RoBERTa, CodeBERTa-small · CodeBERTa

Method

Structural probing (metric/distance probe)

Papers

AST-Probe: Recovering Abstract Syntax Trees from Hidden Representations of Pre-trained Language Models — Hernández López, José Antonio, Weyssow, Martin, Sánchez Cuadrado, Jesús, Sahraoui, Houari2022