TinyLlama

TinyLlama project

Structures found in this family (2)

Representational-similarity trajectory across depth, Dimensional collapse

By model (1)

TinyLlama-1.1B · 1.1B

Representational-similarity trajectory across depth, Dimensional collapse

Observations (2)

Real GPT-2, TinyLlama, and Qwen2.5 representation trajectories show higher curvature for reasoning than lexical tasks, and a universal three-phase layerwise structure

Sequential post-training (SFT, preference optimization, safety/refusal tuning, math/code specialization, long-CoT tuning, chained across stages) progressively compresses real LLMs' hidden-state representations into a low-rank, anisotropic, and homogeneous feature space

Papers

Trajectory Geometry of Transformer Representations Across Layers (2026), Representation Collapse in Sequential Post-Training of Large Language Models (2026)