Chemical Language Model (ChemBERTa / MoLFormer / RoBERTa-Zinc family)

various (academic)

Structures found in this family (2)

Linear Separability, Concept Crystals (Parallelogram/Trapezoid Structure)

By model (3)

MolFormer (100M molecules)

Linear Separability, Concept Crystals (Parallelogram/Trapezoid Structure)

ChemBERTa-2 (77M molecules)

Linear Separability

RoBERTa-Zinc-480M

Linear Separability

Observations (2)

Pretraining improves chemical-LM substructure awareness in upper layers

SMI-TED embeddings obey a linear carbon-chain composition law

Papers

Probing Chemical Language Models: Effects of Pre-training and Fine-tuning (2026), A Large Encoder-Decoder Family of Foundation Models For Chemical Language (2024)