BERT derives local syntax before global; GPT-2 builds both in parallel

measured in 1 paper

Someya et al. apply Derivational Probing (per-layer structural probes plus Tenney et al.'s expected-layer metric) separately to macro- and micro-syntactic subgraphs in BERT-base/large and GPT-2 small/medium [someya-etal-2025] In both BERT models the macro-syntactic subgraph has the highest expected layer, evidence of bottom-up derivation (local structure assembled first, integrated globally later) [someya-etal-2025] GPT-2 shows a more parallel pattern with micro- and macro-syntactic expected layers aligned, so the derivation order is architecture/directionality-dependent [someya-etal-2025] Timing of macro-syntactic integration predicts subject-verb agreement: BERT-base tends to fail when it forms prematurely, BERT-large when it is delayed [someya-etal-2025]

Structure

Tree Metric Embedding

Context

expected layer, scalar mixing, bottom-up vs. top-down derivation, subject-verb agreement

Confirmed in models

BERT-base-cased · BERT, BERT-large-cased · BERT, GPT-2-small · GPT, GPT-2-Medium · GPT

Method

Derivational probing

Papers

Derivational Probing: Unveiling the Layer-wise Derivation of Syntactic Structures in Neural Language Models — Someya, Taiga, Yoshida, Ryo, Yanaka, Hitomi, Oseki, Yohei2025 · arXiv:2506.21861