Diffusion h-space directions arise from PCA, Jacobian SVD, and diff-of-means

measured in 1 paper

Haas et al. find diffusion h-space edit directions via three techniques needing no CLIP or text guidance [haas-etal-2023-interpretable-directions-diffusion] Per-timestep incremental PCA gives unsupervised global directions whose top 2-3 components are semantically dominant (pose, gender, age, smile) [haas-etal-2023-interpretable-directions-diffusion] Per-image Jacobian top singular vectors (via power iteration on J^T J) give locally-optimal directions, a per-sample notion distinct from population PCA [haas-etal-2023-interpretable-directions-diffusion] A difference-of-means vector gives supervised directions, and a linear-projection disentangling step cuts cross-attribute leakage (Glasses-on-Age 0.68 to 0.13) [haas-etal-2023-interpretable-directions-diffusion] Demonstrated on a plain DDPM (CelebA-HQ 256) plus LSUN-church/bedroom, a narrower model range than Asyrp [haas-etal-2023-interpretable-directions-diffusion]

Structure

Linear Direction

Context

h-space, PCA-based global directions, Jacobian spectral analysis, power iteration, difference-of-means supervised direction, linear disentangling projection, cross-attribute leakage

Confirmed in models

DDPM (CelebA-HQ 256x256, HuggingFace Diffusers) · DDPM (Denoising Diffusion Probabilistic Models, HuggingFace Diffusers checkpoints), DDPM (LSUN-Church 256x256, HuggingFace Diffusers) · DDPM (Denoising Diffusion Probabilistic Models, HuggingFace Diffusers checkpoints), DDPM (LSUN-Bedroom 256x256, HuggingFace Diffusers) · DDPM (Denoising Diffusion Probabilistic Models, HuggingFace Diffusers checkpoints)

Method

PCA, Jacobian spectral direction analysis, Difference-in-means direction

Papers

Discovering Interpretable Directions in the Semantic Latent Space of Diffusion Models — Haas, Rene, Huberman-Spiegelglas, Inbar, Mulayoff, Rotem, Grasshof, Stella, Brandt, Sami S., Michaeli, Tomer2023 · arXiv:2303.11073