Fine-tuning reshapes BERT's recoverable syntax tree, differently per task

Perez-Mayos et al. apply the structural probe to layer 7 of BERT-base-cased across checkpoints throughout fine-tuning on six tasks spanning morphology, syntax, and semantics [perez-mayos-etal-2021] Constituency and dependency parsing reinforce tree recoverability early and preserve it, even constituency parsing which is cast as sequence labeling with no explicit tree in its loss [perez-mayos-etal-2021] PoS tagging instead steadily loses tree-distance recoverability, contradicting the fine-tuning-is-conservative conclusion for that case [perez-mayos-etal-2021] The three semantic tasks (SRL, QA, paraphrase) show smaller absolute losses while their relative distance/depth ordering stays comparatively stable [perez-mayos-etal-2021]