Conditional probabilities extracted from real BERT MLMs via cloze prompts recover a formal context (object-attribute incidence) from which a Formal-Concept-Analysis lattice is built, including a concept absent from human ontologies

- Xiong & Staab probe pretrained BERT MLMs (BERT-base, BERT-distill, BERT-large, all uncased) with cloze prompts (e.g. "The [MASK] is an animal that can [MASK]") to read off conditional object-attribute probabilities, argue that the MLM objective implicitly learns a formal context in the Formal Concept Analysis (FCA) sense, and require no training of the MLM. [xiong-staab-2025-tokens-to-lattices] - The extracted probabilities recover the gold formal context far better than a last-layer-embedding-distance baseline (BERT-base on Region-language: MRR 0.823-0.842 vs. 0.340 for BertEmb; Animal-behavior MRR about 0.21 vs. trivial for BertEmb), and are used for multilabel concept classification (Region-language F1 up to 0.709 for BERT-large). [xiong-staab-2025-tokens-to-lattices] - After min-max normalization and binarization, FCA is applied to build a concept lattice (a closure-based partial order of (extent, intent) pairs, a lattice by FCA's construction rather than a separately tested property of the model); the Region-language lattice qualitatively shows a "German and French speaking regions" concept (Switzerland, Luxembourg, Belgium) that is not predefined by a human ontology. The lattice itself is shown qualitatively (Fig. 4b), not scored against a gold lattice. [xiong-staab-2025-tokens-to-lattices] - Domain-specific BioBERT and PharmBERT recover the Disease-symptom formal context and concept classification better than general BERT, which the authors attribute to pretraining-data frequency; the paper does not report disease-symptom lattices in the main text. [xiong-staab-2025-tokens-to-lattices]