Pretraining improves chemical-LM substructure awareness in upper layers

Karnysheva, Klakow & Lee probe eight pretrained chemical language models and six randomly-initialized counterparts for 78 molecular substructures layer-by-layer [karnysheva-etal-2026-probing-chemical-language-models] Pretraining generally improves substructure awareness, particularly in upper layers, while randomly-initialized models already linearly encode ring structures at layer 1 [karnysheva-etal-2026-probing-chemical-language-models] Fine-tuning on two downstream tasks affects task-relevant substructures more than others [karnysheva-etal-2026-probing-chemical-language-models]