CCA / mutual-information layer-trajectory analysis

methods / Theoretical / Analytical / Geometric analysis / CCA / mutual-information layer-trajectory analysis

CCA / mutual-information layer-trajectory analysis

Techniqueintermediate

Tracks how similar a network's own layer representations are to each other (via canonical correlation analysis) and how much information they retain about specific variables (via a mutual-information estimator), across depth — producing a layer-indexed trajectory rather than a single per-layer statistic.

Used in (1 observation)

CCA and mutual-information trajectories differ by training objective across depthmeasured

structure: Representational-similarity trajectory across depth · models: Transformer-base trained on machine translation (En-Ru/En-Fr), Transformer-base trained as a left-to-right language model, Transformer-base trained with a masked-language-modeling objective · paper: The Bottom-up Evolution of Representations in the Transformer: A Study with Machine Translation and Language Modeling Objectives