Fine-tuning widens convex-cluster margins without rearranging the geometry

measured in 1 paper

Zhou & Srikumar reuse DirectProbe on five BERT checkpoints across four linearly-separable tasks to test what fine-tuning does to convex-cluster geometry [zhou-srikumar-2021-finetuning] Fine-tuning consistently increases the minimum inter-cluster margin between every label pair, the geometric mechanism behind accuracy gains [zhou-srikumar-2021-finetuning] A new Spatial Similarity metric shows this is not arbitrary rearrangement: pre/post cluster geometry retains Pearson correlation above 0.5 in higher layers [zhou-srikumar-2021-finetuning] The same metric explains the one exception (BERT-small preposition-supersense accuracy dropping) via the lowest train/test spatial similarity of any condition (0.44) [zhou-srikumar-2021-finetuning]

Structure

Linear Separability

Context

convex hull, fine-tuning dynamics, representational stability, version space

Confirmed in models

BERT-tiny · BERT, BERT-mini · BERT, BERT-small · BERT, BERT-medium · BERT, BERT-base-uncased · BERT

Method

Convex-hull clustering (DirectProbe), Spatial similarity (inter-cluster distance correlation)

Papers

A Closer Look at How Fine-tuning Changes BERT — Zhou, Yichu, Srikumar, Vivek2021 · arXiv:2106.14282