Jacobian spectral direction analysis

methods / Direction Extraction / Jacobian spectral direction analysis

Techniqueadvanced

Finds image-specific (local) semantic edit directions by computing the Jacobian of a generative model's output map with respect to a bottleneck latent, then taking its SVD - the top right singular vectors are the directions that locally perturb the output the most, computed per-sample without any labels or auxiliary loss.

Used in (4 observations)

A pullback Riemannian metric yields diffusion tangent-space directionsmeasured

structure: Linear Direction · models: DDPM (CelebA-HQ 256x256, HuggingFace Diffusers), DDPM+P2-weighting (AFHQ 256x256), DDPM (LSUN-Church 256x256, HuggingFace Diffusers), DDPM (LSUN-Bedroom 256x256, HuggingFace Diffusers), DDPM (LSUN-Cat 256x256), DDPM (LSUN-Horse 256x256), DDPM (ImageNet 256x256), DDPM+P2-weighting (FFHQ 256x256), DDPM+P2-weighting (Flowers 256x256), Stable Diffusion v2.1 · paper: Understanding the Latent Space of Diffusion Models through the Lens of Riemannian Geometry

Diffusion h-space directions arise from PCA, Jacobian SVD, and diff-of-meansmeasured

structure: Linear Direction · models: DDPM (CelebA-HQ 256x256, HuggingFace Diffusers), DDPM (LSUN-Church 256x256, HuggingFace Diffusers), DDPM (LSUN-Bedroom 256x256, HuggingFace Diffusers) · paper: Discovering Interpretable Directions in the Semantic Latent Space of Diffusion Models

A diffusion score-Jacobian splits into tangent and normal subspacesmeasured

structure: Linear Subspace · models: Stable Diffusion v2.1 · paper: Be Tangential to Manifold: Discovering Riemannian Metric for Diffusion Models

StyleGAN2's W space is globally warped; local bases avoid traversal collapsemeasured

structure: Affine Subspace, Linear Subspace · models: StyleGAN2 (trained on FFHQ, 1024x1024) · paper: Do Not Escape From the Manifold: Discovering the Local Coordinates on the Latent Space of GANs