BERT's semantic-sense subspace is largely separate from its syntactic one

Coenen et al. train a cosine-based contrastive probe on BERT-base to find a subspace where same-sense embeddings cluster and different senses separate [coenen-etal-2019] A nearest-centroid classifier in this subspace reaches 71.1-71.5% F1 on word-sense disambiguation, with accuracy holding down to probe dimension m=128 before degrading [coenen-etal-2019] Concatenating two senses of a word shifts the keyword's embedding toward the opposing sense's centroid, and projecting into the sense subspace reduces the induced misclassification (7.31% vs 8.23%) [coenen-etal-2019] The singular values of A^T B (semantic probe A vs Hewitt-Manning syntactic probe B) fall to zero far faster than those of A or B alone, so the semantic and syntactic subspaces are approximately orthogonal (largely separate) [coenen-etal-2019] A separate linear probe on BERT's attention matrices decodes dependency existence (85.8%) and type (71.9%) [coenen-etal-2019]