Back to Subject

Unit 6: Generative Vision Models

CSE471 — Deep Learning For Computer Vision 10 min read

I. Foundations of Generative Vision

Generative vision models learn the statistical structure of visual data so they can create, transform, restore, or relate images to other modalities. Unlike discriminative models, which estimate labels or decision boundaries, generative models represent a data distribution (p_{\text{data}}(x)), a conditional distribution (p(x\mid c)), or a tractable approximation to one.

  • Data distribution: Training images (x) are treated as samples from an unknown distribution (p_{\text{data}}(x)).
  • Latent representation: A compact variable (z), commonly sampled from (p(z)=\mathcal N(0,I)), encodes factors from which an image can be generated.
  • Generator: A neural network (G\theta) maps latent variables or conditions to images, such as (\hat{x}=G\theta(z)).
  • Conditional generation: Additional information (c), such as a class, source image, or text prompt, controls the output through (G(z,c)).
  • Learning objectives: Models may optimize likelihood bounds, adversarial losses, reconstruction errors, perceptual similarity, or contrastive alignment.
  • Evaluation: Output quality includes realism, diversity, semantic correctness, and fidelity to a specified condition.
  • Interpretability: Attribution methods examine which pixels or feature regions influence a model’s prediction.

II. Generative Model Architectures — Learning Visual Distributions

A. Variational autoencoders

A variational autoencoder (VAE) is a probabilistic latent-variable model that learns an encoder (q\phi(z\mid x)) and decoder (p\theta(x\mid z)).

  • Encoder: Given image (x), the encoder predicts vectors (\mu(x)) and (\log \sigma^2(x)) defining a Gaussian approximate posterior:
    TEXT
      q_phi(z | x) = N(z; mu(x), diag(sigma²(x)))

    Here, (\phi) denotes encoder parameters and (z) is the latent code.
  • Reparameterization trick: Sampling is written as (z=\mu+\sigma\odot\epsilon), where (\epsilon\sim\mathcal N(0,I)) and (\odot) is element-wise multiplication. This permits gradients to pass through (\mu) and (\sigma).
  • Objective: Training maximizes the evidence lower bound (ELBO):
    TEXT
      L_ELBO = E_q_phi(z|x)[log p_theta(x|z)]
           - D_KL(q_phi(z|x) || p(z))

    The first term rewards reconstruction; the Kullback–Leibler term regularizes the posterior toward prior (p(z)).
  • Generation: A new image is obtained by sampling (z\sim p(z)) and decoding (x\sim p_\theta(x\mid z)).
  • Trade-off: VAEs provide smooth latent interpolation and stable training, but pixel-based reconstruction losses often produce blurrier images than adversarial models.

B. GAN architectures

A generative adversarial network (GAN) trains a generator (G) against a discriminator (D) in a two-player minimax game.

  • Generator: (G(z)) converts random noise (z\sim p(z)) into a synthetic image.
  • Discriminator: (D(x)\in[0,1]) estimates whether (x) is real rather than generated.
  • Original objective:
    TEXT
      min_G max_D V(D,G)
    = E_x~p_data[log D(x)]
    + E_z~p(z)[log(1 - D(G(z)))]

    (D) maximizes correct classification, while (G) minimizes evidence that generated samples are fake.
  • Practical generator loss: Maximizing (\log D(G(z))) is commonly used instead of minimizing (\log(1-D(G(z)))), because it provides stronger early gradients.
  • Training problems: Mode collapse makes many (z) values produce similar images; discriminator dominance causes weak gradients; oscillation can prevent convergence.
  • Variants: Conditional GANs include condition (c) in both networks, while Wasserstein GANs replace probability classification with a critic estimating distributional discrepancy.

C. DCGAN

Deep Convolutional GAN (DCGAN) adapts adversarial learning to images using convolutional architectural constraints.

  • Generator design: Fractionally strided or transposed convolutions progressively transform a latent vector into a spatial image.
  • Discriminator design: Strided convolutions downsample images without separate pooling layers.
  • Normalization: Batch normalization stabilizes activation distributions, although it is normally omitted from the generator output and discriminator input.
  • Activations: The generator typically uses ReLU internally and tanh at its output; the discriminator uses Leaky ReLU.
  • Representation quality: Latent-vector arithmetic and interpolation can reveal learned visual factors, but DCGAN remains vulnerable to mode collapse and checkerboard artifacts.

D. CycleGAN

CycleGAN performs unpaired translation between domains (X) and (Y), requiring collections of each domain but no aligned image pairs.

  • Bidirectional mapping: Generator (G:X\rightarrow Y) is paired with (F:Y\rightarrow X); discriminators (D_Y) and (D_X) enforce realism in the respective target domains.
  • Adversarial losses: (D_Y) distinguishes real (y) from (G(x)), while (D_X) distinguishes real (x) from (F(y)).
  • Cycle consistency:
    TEXT
      L_cyc = E_x[||F(G(x)) - x||_1]
        + E_y[||G(F(y)) - y||_1]

    The (L_1) norm penalizes failure to reconstruct an input after a round trip.
  • Identity loss: Terms such as (\lVert G(y)-y\rVert_1) help preserve color or structure when an image already belongs to the target domain.
  • Limitation: Cycle consistency does not guarantee semantically correct translation; a model may alter important objects while still reconstructing them through the inverse mapping.

E. StyleGAN

StyleGAN modifies the GAN generator so visual attributes can be controlled at different spatial scales.

  • Mapping network: A multilayer network maps input (z) into an intermediate latent vector (w=f(z)), reducing entanglement with the original sampling distribution.
  • Style modulation: Learned affine transformations of (w) modulate convolutional feature channels. Early layers control coarse pose and layout; later layers influence texture and color.
  • Stochastic detail: Per-layer noise inputs generate local variations such as hair placement or skin texture without changing global identity.
  • Style mixing: Two latent codes can control different layer ranges, combining coarse attributes from one code with fine attributes from another.
  • Later improvements: StyleGAN2 reduces characteristic artifacts using weight modulation and demodulation; StyleGAN3 improves translation and rotation consistency by addressing aliasing.

III. Visual Synthesis Tasks — Creating and Transforming Images

A. Image generation

Image generation produces new samples that resemble a learned image distribution, optionally under user-specified conditions.

  • Unconditional generation: The model samples (x=G(z)) without a class or prompt, so (z) controls the output implicitly.
  • Conditional generation: A label, text embedding, segmentation map, or reference image supplies condition (c) in (G(z,c)).
  • Quality criteria:
    1. Fidelity: Individual images should appear realistic and coherent.
    2. Diversity: Generated samples should cover the modes of the training distribution rather than repeat a small set.
  • FID metric: Fréchet Inception Distance compares Gaussian approximations to real and generated Inception features:
    TEXT
      FID = ||mu_r - mu_g||²
        + Tr(Sigma_r + Sigma_g - 2(Sigma_r Sigma_g)^(1/2))

    Here, (\mu) and (\Sigma) are feature means and covariances; lower FID is generally better.

B. Image-to-image translation

Image-to-image translation maps a source image to a target-domain image while preserving relevant source content.

  • Paired translation: Models such as pix2pix train on aligned pairs ((x,y)), combining an adversarial loss with reconstruction term (\lVert y-G(x)\rVert_1).
  • Unpaired translation: CycleGAN learns from separate source and target collections using adversarial and cycle-consistency objectives.
  • Structured conditions: Inputs may be edge maps, semantic layouts, depth maps, sketches, daytime scenes, or medical scans.
  • Core constraint: The output must balance target-domain realism with source fidelity; optimizing only realism can change geometry or semantic identity.

C. Super-resolution

Super-resolution reconstructs a high-resolution image (I{HR}) from a degraded low-resolution observation (I{LR}).

  • Degradation model: A common formulation is (I{LR}=(I{HR}*k)\downarrow_s+n), where (k) is a blur kernel, (\downarrow_s) is downsampling by scale (s), and (n) is noise.
  • Pixel loss: (L_1) or mean-squared error promotes high peak signal-to-noise ratio but tends to average plausible details.
  • Perceptual loss: Feature differences (\lVert\phi(\hat I)-\phi(I_{HR})\rVert), using a pretrained network (\phi), better preserve perceived texture and structure.
  • Adversarial loss: SRGAN-style systems add a discriminator to produce sharper textures, although generated detail may be plausible rather than factually present.
  • Risk: In medical, forensic, and scientific images, hallucinated detail must not be treated as recovered evidence.

IV. Vision–Language Representation — Connecting Images and Text

A. CLIP for image-text alignment

Contrastive Language–Image Pre-training (CLIP) learns a shared embedding space by matching images with their associated natural-language descriptions.

  • Dual encoders: An image encoder produces (v_i), while a text encoder produces (t_j); both vectors are normalized.
  • Similarity: Alignment is measured with scaled cosine similarity:
    TEXT
      s_ij = (v_i · t_j) / tau

    Here, (s_{ij}) is image–text similarity and (\tau) is a learned temperature.
  • Contrastive objective: Within a batch, the correct image–caption pair is the positive example and other combinations are negatives. Cross-entropy is optimized in both image-to-text and text-to-image directions.
  • Zero-shot classification: Class prompts such as “a photo of a dog” are embedded, and an image receives the class whose text embedding has highest cosine similarity.
  • Generative use: CLIP similarity can guide generated images toward a prompt, evaluate semantic alignment, or support retrieval.
  • Limitations: Results depend on prompt wording and training-data biases; similarity does not prove factual correctness or detailed spatial understanding.

V. Model Explanation — Inspecting Visual Decisions

A. Interpretability techniques

Interpretability techniques seek evidence about how input features, internal representations, or training examples affect model outputs.

  • Post-hoc explanations: Attribution is computed after training without changing the predictor, as in saliency maps, Grad-CAM, occlusion, and integrated gradients.
  • Intrinsic inspection: Feature visualization, activation maximization, and latent traversal examine what neurons, channels, or latent dimensions represent.
  • Perturbation methods: Regions are masked or altered and the change in class score (f_c(x)) is measured; strong score reduction suggests relevance.
  • Evaluation criteria: Explanations should be faithful to model behavior, stable under small irrelevant changes, and localized meaningfully.
  • Caution: A visually convincing explanation is not necessarily causal; correlated pixels may receive importance even when they do not represent the true concept.

B. Grad-CAM

Gradient-weighted Class Activation Mapping (Grad-CAM) localizes class-relevant regions using gradients flowing into a convolutional layer.

  • Channel weights: For class score (y^c) and feature map (A^k), gradients are spatially averaged:
    TEXT
      alpha_k^c = (1/Z) sum_i sum_j (partial y^c / partial A_ij^k)

    Here, (k) indexes channels, (i,j) index positions, and (Z) is the number of spatial locations.
  • Heatmap:
    TEXT
      L_Grad-CAM^c = ReLU(sum_k alpha_k^c A^k)

    ReLU retains features that positively support class (c).
  • Interpretation: The low-resolution heatmap is upsampled and overlaid on the input, revealing regions associated with the selected prediction.
  • Limitation: Localization is coarse because deep feature maps have low spatial resolution, and the map indicates model attention rather than object boundaries or causation.

C. Saliency maps

Saliency maps estimate each input pixel’s influence on a selected output using the output gradient with respect to the image.

  • Definition: For class score (S_c(x)), basic saliency is:
    TEXT
      M = |partial S_c(x) / partial x|

    Here, (x) is the input image and (M) contains absolute gradient magnitudes.
  • Meaning: A large value indicates that a small change to that pixel could strongly alter the class score locally.
  • Channel reduction: For RGB input, absolute gradients may be maximized or averaged across the three color channels to obtain one value per pixel.
  • Variants: SmoothGrad averages maps from noisy copies of (x); integrated gradients accumulates gradients from a baseline (x') to the actual input.
  • Limitations: Raw maps are often noisy, sensitive to gradient saturation, and unstable under small perturbations; they identify sensitivity rather than proving semantic importance.
Previous
Unit 5