MATH · IN · MODELS
structures / Manifolds / Concept lattice (Formal Concept Analysis half-space model)

Concept lattice (Formal Concept Analysis half-space model)

OBJECTcombinatorial-objectadvancedhow it's classified →

Each attribute is a linear direction defining an origin-passing half-space; a concept (a set of co-occurring attributes) is the polyhedral cone formed by intersecting those half-spaces. The set of all such concepts, ordered by inclusion, forms a complete lattice with well-defined meet (intersection) and join (least common generalization) operations — recovering Formal Concept Analysis directly from embedding geometry.

Replicationcomputed from the corpus — never hand-assigned
2 papers · no shared authors2 architecture classes · across papers1 domain4 model families · across papers
Filled = two or more values reported by papers that share no author — replication. Outlined = two or more values, but all from a single study — breadth, not replication. Grey = a single value. Derived from paper authorship and each model's architecture class, domain and family; it updates itself when a paper is added.

Definition

For attributes m∈Mm\in M with directions dm∈Rd\mathbf d_m\in\mathbb R^d, a concept defined by attribute subset Y⊆MY\subseteq M occupies the region

R(Y)={v∈Rd:v⋅dm≥0 for all m∈Y},\mathcal R(Y) = \{\mathbf v\in\mathbb R^d : \mathbf v\cdot\mathbf d_m \geq 0\ \text{for all}\ m\in Y\},

a polyhedral cone in the sense of Cone (an H-representation — intersection of half-spaces — rather than the V-representation of non-negative ray combinations, but the same underlying object class). Pairing each region with the object embeddings it contains recovers a formal concept (extent, intent) in the sense of Formal Concept Analysis (FCA): the set of formal concepts, ordered by extent inclusion (equivalently, reverse intent inclusion), forms a complete lattice — every subset of concepts has a greatest lower bound (meet, satisfying the union of both concepts’ attributes, A∧B=R(YA∪YB)A\wedge B = \mathcal R(Y_A\cup Y_B)) and a least upper bound (join, the smallest common generalization, approximated by the conic hull of both concepts’ attribute directions).

Relative to a single cone or polytope

A single Cone is one region; a single Polytope (Simplex) is one convex hull of mutually-exclusive category values. A concept lattice is a system of many such cone-regions simultaneously, connected by a Galois connection between an object set and an attribute set (the same object↔attribute duality used throughout FCA), so that the regions are not independent shapes but form a provably complete partial order with algebraic meet/join structure. This is what licenses treating LLM representations as encoding a full symbolic hierarchy (subsumption, generalization, refinement), not just isolated categorical contrasts.

Key evidence

Xiong (2026) proves (Theorem 1) that thresholded linear attribute directions, softened via a logistic incidence function Pα(m(g)=1)=σ(α(vg⋅ℓˉm−τm))P_\alpha(m(g)=1)=\sigma(\alpha(\mathbf v_g\cdot\bar\ell_m-\tau_m)) and binarized at a confidence level δ\delta, induce a binary incidence relation whose Galois-connection closure is provably a complete lattice — and (Proposition 1) that per-attribute thresholds can always be absorbed into a single global embedding shift, reducing the general thresholded model to an origin-passing canonical form. Empirically, across five WordNet sub-hierarchies (three physical domains — animal, plant, food — and two abstract domains — event, cognition) and three models (LLaMA-3.1-8B, Gemma-7B, Mistral-7B), LDA-estimated attribute directions recover the ground-truth object–attribute formal context with F1 above 78% on physical domains, and projection-profile-based subsumption scoring recovers WordNet’s hypernym partial order (up to F1 77.1). Meet and join operations computed directly on attribute projection profiles return WordNet-plausible refinements and generalizations (e.g. join(dog, wolf) → predator/canine/hunter; meet(horse, zebra) → pony/stallion/foal). See Formal Concept Analysis (FCA) lattice construction for the construction method.

How to detect it

Fit a linear direction and separating threshold for each of several binary attributes, form the induced object–attribute incidence relation, and check (via the Galois-connection closure) whether the resulting formal concepts form a complete lattice matching an independently-known symbolic hierarchy (e.g. WordNet hypernymy) — rather than checking only whether individual attributes are separately linearly decodable.

A mechanistically distinct route to the same hierarchical-geometry claim: spectral eigenvectors of co-occurrence statistics

Nava & Wyart (2026) reach a structurally similar conclusion — that taxonomic (“is-a”) hierarchy is legible in embedding geometry — via a different mechanism than the half-space/lattice construction above: rather than thresholded linear attribute directions forming a Galois-connection lattice, they show the leading eigenvectors of a word co-occurrence Gram matrix directly split broad-to-fine taxonomic branches (top-k eigenspace alignment far exceeding a shuffled baseline, with an exponentially decaying alignment-vs-depth kernel), extending from word2vec embeddings to Gemma-2B unembeddings and Llama-3.2-1B internal activations. A parent-child “innovation”-vector diagnostic (child-minus-parent direction near-orthogonal to the parent direction) recovers the same near-zero-cosine linear-representation signature used elsewhere in this map, without invoking the cone/half-space/FCA apparatus. See leading-eigenvectors-of-a-word-cooccurrence-gram-matrix-separate-broad-to-fine-taxonomic-concept-branches-and-the-same-spectral-tree-geometry-appears-in-gemma-and-llama-activations.

Key papers

  • Ganter & Wille (1999); Ganter, Stumme & Wille (2005). Formal Concept Analysis. — origin of FCA itself, prior to any connection to LLMs.
  • Xiong & Staab (2025). From Tokens to Lattices: Emergent Lattice Structures in Language Models. — first connection of FCA to language models, limited to masked language models.
  • Xiong (2026). The Lattice Representation Hypothesis of Large Language Models. arXiv:2603.01227 — unifies the Linear Representation Hypothesis with FCA via half-space intersections; origin of the half-space/lattice construction described here.

Found in (2 observations · 4 families)

Llama

The Lattice Representation Hypothesis of Large Language Models (2026)measured

LLM attribute half-spaces intersect into a complete concept lattice

Details

- The Lattice Representation Hypothesis models each binary attribute (e.g. "can fly") as a linear direction with a threshold, so a concept defined by co-occurring attributes is the convex polyhedral cone formed by intersecting their origin-passing half-spaces. [xiong-2026] - Theorem 1 proves the Galois closure of the softened, thresholded object-attribute relation is a complete lattice, and Proposition 1 absorbs per-attribute thresholds into one global embedding shift (a canonical origin-passing form). [xiong-2026] - Across five WordNet sub-hierarchies in LLaMA-3.1-8B, Gemma-7B and Mistral-7B, LDA/Fisher attribute directions recover the ground-truth formal context at F1 above 78% on physical domains (best Gemma-7B 83.2% on Animal) versus 53-68% mean-embedding and 45-48% random baselines. [xiong-2026] - Projection-profile subsumption scoring recovers WordNet's hypernym partial order (up to F1 77.1, LLaMA on Animal), and soft meet/join operators beat baselines at retrieving the true lowest-common-subconcept / least-common-hypernym (Table 3). [xiong-2026] - A scaling comparison across LLaMA-3 models (3B, 8B, 70B) shows scaling helps abstract domains far more than physical ones; observational, no intervention. [xiong-2026]

models: Llama-3.1-8B, LLaMA-3 3B, Llama-3-8B, Llama-3-70B · method: Linear probing, Formal Concept Analysis (FCA) lattice construction

Gemma

The Lattice Representation Hypothesis of Large Language Models (2026)measured

LLM attribute half-spaces intersect into a complete concept lattice

Details

- The Lattice Representation Hypothesis models each binary attribute (e.g. "can fly") as a linear direction with a threshold, so a concept defined by co-occurring attributes is the convex polyhedral cone formed by intersecting their origin-passing half-spaces. [xiong-2026] - Theorem 1 proves the Galois closure of the softened, thresholded object-attribute relation is a complete lattice, and Proposition 1 absorbs per-attribute thresholds into one global embedding shift (a canonical origin-passing form). [xiong-2026] - Across five WordNet sub-hierarchies in LLaMA-3.1-8B, Gemma-7B and Mistral-7B, LDA/Fisher attribute directions recover the ground-truth formal context at F1 above 78% on physical domains (best Gemma-7B 83.2% on Animal) versus 53-68% mean-embedding and 45-48% random baselines. [xiong-2026] - Projection-profile subsumption scoring recovers WordNet's hypernym partial order (up to F1 77.1, LLaMA on Animal), and soft meet/join operators beat baselines at retrieving the true lowest-common-subconcept / least-common-hypernym (Table 3). [xiong-2026] - A scaling comparison across LLaMA-3 models (3B, 8B, 70B) shows scaling helps abstract domains far more than physical ones; observational, no intervention. [xiong-2026]

models: Gemma-7B · method: Linear probing, Formal Concept Analysis (FCA) lattice construction
Hierarchical Concept Geometry in Language Models Emerges from Word Co-occurrence (2026)measured

Word-cooccurrence eigenvectors form a spectral tree recurring in Gemma

Details

The successive leading eigenvectors of a normalized word co-occurrence (PMI-related) Gram matrix progressively separate taxonomic branches from broad to fine, a coarse-to-fine spectral splitting tree [nava-wyart-2026-hierarchical-concept-geometry-in-language-models-emerges-from-word-cooccurrence] Top-k eigenspace alignment far exceeds a shuffled baseline and fits an exponential decay f(d)=1.967*e^(-1.235*d) in WordNet distance d [nava-wyart-2026-hierarchical-concept-geometry-in-language-models-emerges-from-word-cooccurrence] The same spectral-tree geometry, first validated in word2vec, recurs in Gemma-2B unembeddings and in the mid-layer residual stream of Gemma-2B [nava-wyart-2026-hierarchical-concept-geometry-in-language-models-emerges-from-word-cooccurrence] Parent-to-child innovation vectors are near-orthogonal to the parent (a linear-representation signature); the geometry is derived as a prediction of a co-occurrence model, not tested causally [nava-wyart-2026-hierarchical-concept-geometry-in-language-models-emerges-from-word-cooccurrence]

models: Gemma-2B · method: Spectral analysis of the co-occurrence/PMI matrix, Direction Extraction, Geometric analysis

Mistral

The Lattice Representation Hypothesis of Large Language Models (2026)measured

LLM attribute half-spaces intersect into a complete concept lattice

Details

- The Lattice Representation Hypothesis models each binary attribute (e.g. "can fly") as a linear direction with a threshold, so a concept defined by co-occurring attributes is the convex polyhedral cone formed by intersecting their origin-passing half-spaces. [xiong-2026] - Theorem 1 proves the Galois closure of the softened, thresholded object-attribute relation is a complete lattice, and Proposition 1 absorbs per-attribute thresholds into one global embedding shift (a canonical origin-passing form). [xiong-2026] - Across five WordNet sub-hierarchies in LLaMA-3.1-8B, Gemma-7B and Mistral-7B, LDA/Fisher attribute directions recover the ground-truth formal context at F1 above 78% on physical domains (best Gemma-7B 83.2% on Animal) versus 53-68% mean-embedding and 45-48% random baselines. [xiong-2026] - Projection-profile subsumption scoring recovers WordNet's hypernym partial order (up to F1 77.1, LLaMA on Animal), and soft meet/join operators beat baselines at retrieving the true lowest-common-subconcept / least-common-hypernym (Table 3). [xiong-2026] - A scaling comparison across LLaMA-3 models (3B, 8B, 70B) shows scaling helps abstract domains far more than physical ones; observational, no intervention. [xiong-2026]

models: Mistral-7B · method: Linear probing, Formal Concept Analysis (FCA) lattice construction

word2vec

Hierarchical Concept Geometry in Language Models Emerges from Word Co-occurrence (2026)measured

Word-cooccurrence eigenvectors form a spectral tree recurring in Gemma

Details

The successive leading eigenvectors of a normalized word co-occurrence (PMI-related) Gram matrix progressively separate taxonomic branches from broad to fine, a coarse-to-fine spectral splitting tree [nava-wyart-2026-hierarchical-concept-geometry-in-language-models-emerges-from-word-cooccurrence] Top-k eigenspace alignment far exceeds a shuffled baseline and fits an exponential decay f(d)=1.967*e^(-1.235*d) in WordNet distance d [nava-wyart-2026-hierarchical-concept-geometry-in-language-models-emerges-from-word-cooccurrence] The same spectral-tree geometry, first validated in word2vec, recurs in Gemma-2B unembeddings and in the mid-layer residual stream of Gemma-2B [nava-wyart-2026-hierarchical-concept-geometry-in-language-models-emerges-from-word-cooccurrence] Parent-to-child innovation vectors are near-orthogonal to the parent (a linear-representation signature); the geometry is derived as a prediction of a co-occurrence model, not tested causally [nava-wyart-2026-hierarchical-concept-geometry-in-language-models-emerges-from-word-cooccurrence]

models: word2vec · method: Spectral analysis of the co-occurrence/PMI matrix, Direction Extraction, Geometric analysis