OpenAI Embeddings

OpenAI

Structures found in this family (4)

Linear Direction, Linear Subspace, Circle, 1D continuum manifold

By model (2)

text-embedding-3-large

Linear Direction, Circle, 1D continuum manifold

text-embedding-3-small

Linear Subspace, Linear Direction

Observations (4)

Analogical relations fit a rotational operator better than an additive offset

Independently-trained embedding models share linear structure (CKA 0.60-0.88)

SAE decoder cosine reveals feature splitting in dense embeddings and steers search

Papers

Mapping Semantic & Syntactic Relationships with Geometric Rotation in Embedding Space (2025), Characterizing Linear Alignment Across Language Models (2026), The Origins of Representation Manifolds in Large Language Models (2025), Disentangling Dense Embeddings with Sparse Autoencoders (2024)