GraphCodeBERT

Microsoft

Structures found in this family (1)

Tree Metric Embedding

By model (1)

GraphCodeBERT-base · 125M

Tree Metric Embedding

Observations (2)

AST-Probe recovers full code syntax trees from a 64-128 dimensional subspace

A Hewitt-Manning-style distance probe recovers AST tree distances from CodeBERT and GraphCodeBERT, peaking at layer 5

Papers

AST-Probe: Recovering Abstract Syntax Trees from Hidden Representations of Pre-trained Language Models (2022), What Do They Capture? - A Structural Analysis of Pre-Trained Language Models for Source Code (2022)