LLM pretraining builds a reusable manifold that a frozen linear probe can decode into realistic time-series trajectories

- Roger et al. (2026) repurpose Qwen3-0.6B as a probabilistic time-series forecaster (forecasting cast as next-token prediction over 1,024 discretized bins) and, separately, fit a single linear map on the concatenated frozen hidden states of all 28 layers (D = 28,672) of the language-pretrained model for 1,920 WikiText-103 sequences, trained by EM-style nearest-neighbor matching to a bank of 10,000 real time series without paired supervision: the decoded outputs' nearest neighbors span 686 distinct real series (36% of predictions), versus 54 (2.8%) with random weights and 4 (0.2%) with random-token inputs through pretrained weights (Sec. 4.1, Fig. 3). Retrieving a WikiText projection that matches the first half of a real series gives MSE 1.91 vs 2.27 for last-value carry-forward on 500 series, winning in only 37% of cases (Sec. 4.2). [roger-etal-2026-llm-pretraining-generalizable-manifold] - Across four adaptation regimes (full finetuning, IO-only, LoRA-attention, LoRA-attention+IO), language-pretrained initialization converges faster and to lower forecasting error than random initialization (Fig. 2; effective data transfer D_T = 6.8 / 2.3 / 5.2 / 3.4 for full / IO-only / LoRA-attention / LoRA-attention+IO), and per-sample gradients are already aligned from step one (mean pairwise cosine over 32 held-out series) while the randomly initialized model stays near zero until step 64-256 (Fig. 5). The authors interpret this as a reusable pretrained manifold but state they do not isolate it from initialization statistics (Sec. 6). [roger-etal-2026-llm-pretraining-generalizable-manifold] - Tracking the effective rank of hidden-state covariance (Sec. 5.2, Fig. 7): the randomly initialized model starts near-isotropic (~440) and collapses to ~10 within 500 steps, whereas the language-pretrained model declines from ~50 to ~27 on time-series inputs, with middle layers (~8-17) increasing in effective rank; on text inputs its effective rank drops from ~165 to ~25 (85% reduction) as perplexity rises. [roger-etal-2026-llm-pretraining-generalizable-manifold]