Time-series foundation models linearly encode concepts and steer, including ECG

measured in 1 paper

Wilinski et al. find block-like redundancy usable for pruning and linearly-represented concepts (periodicity, trends) in Chronos, MOMENT, and Moirai-1.1-R Large [wilinski-etal-2024-exploring-representations] A steering matrix is computed as the difference-in-medians of activation between concept classes, with the paper noting the mean-based variant works equivalently [wilinski-etal-2024-exploring-representations] Adding it steers outputs toward concept-informed predictions on synthetic and forecasting tasks [wilinski-etal-2024-exploring-representations] On a real ECG5000 dataset, MOMENT-plus-SVM samples initially classified 100% as normal all swap output class after steering, a causal classification flip [wilinski-etal-2024-exploring-representations]

Structure

Linear Direction

Context

diff-in-median steering vector per layer (periodic/trending vs. constant synthetic series), Fisher Linear Discriminant Ratio localizing concept strength across layers and tokens, additive steering intervention h_i <- h_i + lambda*S_i, 100% (30/30) causal classification-flip on real ECG5000 data after steering, separate CKA-based block-redundancy finding motivating layer pruning

Confirmed in models

Chronos · Chronos, MOMENT · MOMENT, Moirai-1.1-R Large · Moirai

Method

Difference-in-means direction, Linear probing, Activation Steering (Addition)

Papers

Exploring Representations and Interventions in Time Series Foundation Models — Wiliński, Michał, Goswami, Mononito, Potosnak, Willa, Żukowska, Nina, Dubrawski, Artur2024 · arXiv:2409.12915