DeBERTa

Microsoft

Structures found in this family (2)

Linear Direction, Affine Subspace

By model (2)

DeBERTa-v2-xxlarge · 1.5B

Affine Subspace

DeBERTa-xxlarge-v2-MNLI · 1.5B

Linear Direction

Observations (3)

Linear Artificial Tomography reads and controls concepts across LLMs

Geographic representations are causally used for prediction, not just decodable

Contrast-Consistent Search finds a truth-like direction with no labels

Papers

Representation Engineering: A Top-Down Approach to AI Transparency (2023), More than Correlation: Do Large Language Models Learn Causal Representations of Space? (2023), Discovering Latent Knowledge in Language Models Without Supervision (2023)