DirectProbe finds convex label clusters whose geometry predicts accuracy

Zhou & Srikumar introduce DirectProbe, agglomerative clustering that merges same-label points only while their convex hulls stay disjoint from other labels, testing linear separability without training a classifier [zhou-srikumar-2021] Across BERT, RoBERTa, and ELMo on five linguistic tasks, the number of convex clusters matches the label count exactly when the representation is linearly separable (e.g. 17 clusters for 17-label POS) [zhou-srikumar-2021] The minimum inter-cluster margin tracks best achievable classifier accuracy across layers and fine-tuning, and a training-free nearest-centroid pseudo-classifier correlates with true accuracy [zhou-srikumar-2021] Classifier errors concentrate on label pairs whose clusters sit at small geometric distance (97.17% of one task's errors in the smallest-distance bin) [zhou-srikumar-2021]