Definition

For a binary (or more generally class-partitioned) semantic label $z$ and a diffusion model’s noisy state $x_t$, the conditional entropy $H(z\mid x_t) = -\int p(x_t)\sum_z P(z\mid x_t)\log_2 P(z\mid x_t)\,dx_t$ quantifies how much residual uncertainty about $z$ remains at noise level $t$, using the Bayes-optimal posterior $P(z\mid x_t)$ recovered from the diffusion model’s own class-conditional and unconditional noise predictions via a classifier-free-guidance-style Bayesian update. Its time-derivative $\dot H(z\mid x_t)$ — “bits transferred per unit diffusion time” — is interpreted as the rate of semantic information production: how quickly the generative dynamics resolve which class the eventual sample will belong to, as a function of where in the noise schedule the model currently sits.

Intuition

Early in the reverse (denoising) process $x_t$ is nearly pure noise, so $P(z\mid x_t)$ is close to the class prior and $H(z\mid x_t)$ is near maximal; by the end $x_t\approx x_0$ and $z$ is essentially determined, so $H(z\mid x_t)\to 0$. The interesting question is not the endpoints but the shape of the transition in between: is class identity resolved gradually, or does it snap into place within a narrow intermediate window? A peaked $\dot H$ answers this directly, without requiring any assumption about the geometry of the score function or the data manifold’s curvature — it is purely a statement about how much class-relevant information the noisy state carries, extracted using only the trained model’s own predictions as a Bayesian estimator.

Relative to manifold-curvature-profile

Curvature profile of the representation manifold (including its score-Jacobian spectral-gap and Critical-Boundary-Detector entries) measures geometric properties of the score field itself — eigenvalue spectra, Jacobian norms — along the same noise axis. This node instead measures a purely information-theoretic quantity (a class-conditional entropy rate) computed from the model’s posterior predictions, with no reference to the score function’s Jacobian, curvature, or spectral structure. The two diagnostics can in principle be computed on the same trajectory and compared, but neither reduces to nor is derived from the other.

Properties

• Reduces to a Jensen-Shannon divergence in the two-class, equal-prior case. When $P(z{=}0)=P(z{=}1)=0.5$, $H(z\mid x_t)$ is proportional to the negative Jensen-Shannon divergence between the two class-conditional noise distributions at that noise level — giving the entropy-rate construction a concrete information-theoretic interpretation beyond the general definition.
• Extracted from the model’s own predictions, not an external classifier. The Bayes-optimal posterior is estimated purely from the trained diffusion model’s own conditional and unconditional noise-prediction outputs (following a classifier-free-guidance-style Bayesian update), so the diagnostic is intrinsic to the generative model rather than requiring a separately trained auxiliary classifier.
• The transition window’s location depends on how much structure the two classes share. Handke, Koulischer, Raya & Ambrogioni (2025) find that class pairs with less shared visual structure (e.g. deer vs. car) show semantic information production peaking earlier in the reverse process than pairs sharing more structure (e.g. deer vs. bird, deer vs. cat), on real class-conditional DDPMs trained from scratch on CIFAR-10 — the entropy-rate trajectory’s shape is class-pair-specific, not a universal curve. See handke-etal-2025-class-pair-semantic-information-production-peaks-earlier-for-less-visually-similar-cifar-10-class-pairs-in-real-class-conditional-ddpms.

Key papers

• Handke, F., Koulischer, F., Raya, G. & Ambrogioni, L. (2025). Measuring Semantic Information Production in Generative Diffusion Models. arXiv:2506.10433 — origin of the class-conditional entropy-rate diagnostic, validated on real class-conditional DDPMs trained from scratch on CIFAR-10.