A frozen proto-value subspace is competitive with DQN at ~13x fewer steps
measured in 1 paperFarebrother et al. train an Impala-CNN encoder on offline RL Unplugged Atari (46 games) to estimate the successor measure, proving its features span the top singular vectors (graph-Laplacian eigenvectors / proto-value functions) [farebrother-etal-2023-proto-value-networks] Freezing this encoder and training only a linear RL head online is competitive with DQN using 3.75M agent steps versus DQN's 50M steps (200M frames), about 13x fewer interactions (not 50x) [farebrother-etal-2023-proto-value-networks] Per-game score deltas are large (e.g. Asterix 15,401 vs 2,953; DemonAttack 78,671 vs 4,428; Pong 20.1 vs 11.5) [farebrother-etal-2023-proto-value-networks] The subspace is a designed spectral objective (proto-value functions), imposed by construction rather than freely emergent [farebrother-etal-2023-proto-value-networks]