A real contrastive world model linearly decodes which object changed despite no imposed object slots

Saanum, Schulze Buschoff, Dayan & Schulz train real CNN-encoder Contrastive World Models (CWM) via next-state prediction on rendered/simulated multi-object video (Cubes, 3-body physics, Multi-dSprites, MOVi-simple, MOVi-A), taking the activation difference |encoder(x_t) minus encoder(x_t+1)| between frame pairs where exactly one object changed, and training an L1-regularized linear classifier on this difference vector to predict which object changed [saanum-etal-2024-next-state-prediction-entangled-compositional-object-representations] The linear classifier reaches near-100% held-out accuracy on Cubes and 3-body physics and about 70% (well above chance) on the harder Multi-dSprites and MOVi datasets, while untrained/random encoders and a static (non-predictive) contrastive baseline score near chance, confirming next-state prediction specifically drives the effect rather than architecture alone [saanum-etal-2024-next-state-prediction-entangled-compositional-object-representations] Despite this linear object-decodability, cosine-similarity clustering of the same difference vectors shows the distributed CWM representation is not factorized into clean per-object slots the way an explicit slot-based baseline (CSWM) is -- shared populations of units encode transformation type across different objects -- and a frozen CWM representation can still be decoded into slot-like object codes via a downstream Slot-Attention decoder, showing the compositional structure is recoverable though entangled rather than architecturally imposed [saanum-etal-2024-next-state-prediction-entangled-compositional-object-representations]