Protein-LM probes recover structure; attention heads align above background

Vig et al. analyze five protein language models (TapeBert, ProtBert, ProtBert-BFD, ProtAlbert, ProtXLNet) with linear single-layer probes and attention-head analysis [vig-etal-2020-bertology-meets-biology-interpreting-attention-in-protein-language-models] Individual attention heads concentrate on 3D contact maps, binding sites, and secondary structure far above background (up to 63.2% of a head's attention on contacts vs a 1.3% background; one head 49% on binding sites, Bonferroni p<0.00001) [vig-etal-2020-bertology-meets-biology-interpreting-attention-in-protein-language-models] Linear probes on output embeddings recover the same structures, quantified via precision@L/5 for contacts, precision@L/20 for binding sites, and F1 for secondary structure [vig-etal-2020-bertology-meets-biology-interpreting-attention-in-protein-language-models]