Unit 6: Generative Vision Models - Subjective Questions

A Variational Autoencoder (VAE) is a probabilistic generative model that learns a distribution over latent variables and uses samples from that distribution to generate data.

Main components:

• The encoder maps an input to the parameters and of a latent distribution.
• A latent vector is sampled from .
• The decoder reconstructs or generates an output from .

Difference from a conventional autoencoder:

• A conventional autoencoder maps each input to a deterministic latent vector.
• A VAE maps each input to a probability distribution in latent space.
• A VAE regularizes the latent space so that nearby latent points tend to generate similar outputs.
• New samples can be generated by drawing from the prior and passing it through the decoder.

Thus, a VAE is suitable for both representation learning and controlled data generation.

For an observation and latent variable , the marginal likelihood is

Introduce an approximate posterior :

Applying Jensen's inequality gives

Using , the ELBO becomes

Interpretation:

• The first term is the reconstruction term, which encourages the decoder to reproduce the input.
• The second term is the KL-divergence regularizer, which encourages the approximate posterior to remain close to the prior, usually .

Training maximizes the ELBO, or equivalently minimizes

The encoder of a VAE commonly predicts the mean and standard deviation of a Gaussian approximate posterior:

Directly sampling from this distribution would make ordinary backpropagation through the sampling operation difficult. The reparameterization trick rewrites the sample as

Why it is required:

• It separates the random variable from the trainable encoder parameters.
• The value of becomes a differentiable function of and .
• Gradients can therefore flow from the decoder loss through to the encoder.
• It enables stochastic gradient optimization of the ELBO.

The symbol denotes element-wise multiplication.

A GAN consists of two competing neural networks:

• The generator transforms a random latent vector into a synthetic sample .
• The discriminator estimates whether an input is real or generated.

The original minimax objective is

Training process:

1. Sample real images from the training data.
2. Generate fake images from random latent vectors.
3. Update to assign high probability to real images and low probability to generated images.
4. Update so that generated images are classified as real by .
5. Alternate these updates until the generated distribution approximates the data distribution.

In practice, the generator is often trained using the non-saturating loss because it provides stronger gradients early in training.

GAN training is difficult because it involves a dynamic competition between two networks.

Major difficulties and remedies:

• Mode collapse: The generator produces only a limited variety of outputs. Remedies include minibatch discrimination, feature matching, Wasserstein objectives, and diversity regularization.
• Vanishing gradients: An overly accurate discriminator gives weak gradients to the generator. Non-saturating loss, Wasserstein loss, or balanced update schedules can help.
• Training oscillation: The generator and discriminator may fail to converge. Learning-rate tuning, regularization, and normalization improve stability.
• Exploding gradients: Gradient clipping or gradient penalties can control unstable updates.
• Discriminator overfitting: Data augmentation, dropout, spectral normalization, and limiting discriminator updates are useful.
• Evaluation difficulty: Visual quality alone is subjective, so measures such as FID and Inception Score are used along with human evaluation.

A stable GAN generally requires balanced model capacities, careful optimization, adequate data, and continuous monitoring of both quality and diversity.

Deep Convolutional GAN (DCGAN) adapts GANs to image generation using convolutional neural networks.

Generator design:

• Begins with a random latent vector.
• Uses learned upsampling through transposed convolutions or equivalent upsampling blocks.
• Applies batch normalization in most hidden layers.
• Uses ReLU activations internally and typically at the output.

Discriminator design:

• Uses strided convolutions for downsampling instead of pooling layers.
• Applies Leaky ReLU activations.
• Commonly uses batch normalization except at the input or output where inappropriate.
• Produces a real-versus-fake prediction.

Important guidelines:

• Replace fully connected hidden layers with convolutional operations.
• Avoid deterministic pooling when learned downsampling is possible.
• Normalize intermediate activations to stabilize optimization.
• Scale training images to match the generator's output activation range, such as for .

These choices exploit spatial structure and generally produce more coherent images than a multilayer-perceptron GAN.

Objective:

• A VAE maximizes the ELBO, combining reconstruction accuracy with latent-distribution regularization.
• A GAN uses adversarial training in which a generator competes with a discriminator.

Latent space:

• VAEs explicitly encourage a smooth, structured latent space close to a chosen prior.
• GAN latent spaces can be semantically meaningful but are not regularized through an encoder in the basic formulation.

Output quality:

• VAEs often generate diverse but relatively smooth or blurry images because of likelihood-based reconstruction losses.
• GANs commonly produce sharper and more realistic images but may omit modes of the data distribution.

Training behavior:

• VAEs are generally stable and optimize a single well-defined bound.
• GANs can suffer from mode collapse, oscillation, and sensitivity to hyperparameters.

Inference:

• A VAE naturally provides an encoder for mapping an image to a latent representation.
• A basic GAN has no inverse mapping unless an encoder or latent-inversion method is added.

Therefore, VAEs are useful when latent inference and coverage are important, while GANs are often preferred for high-fidelity synthesis.

CycleGAN learns mappings between two visual domains and without requiring paired examples. It uses:

• A generator .
• A generator .
• A discriminator for domain .
• A discriminator for domain .

Each generator has an adversarial objective. For example,

The cycle-consistency loss requires a translated image to return to its original form:

An optional identity loss discourages unnecessary changes:

The total objective combines adversarial, cycle-consistency, and identity terms with weighting coefficients. This makes translations resemble the target domain while preserving important source content.

Paired image-to-image translation:

• Requires aligned source and target images representing the same scene.
• A direct reconstruction loss such as can compare the generated output with its ground truth.
• Pix2Pix is a common paired translation model.
• Applications include label-map-to-photo synthesis, edge-to-image conversion, and supervised colorization.

Unpaired image-to-image translation:

• Uses independent collections of images from two domains without one-to-one correspondence.
• Direct pixel-level supervision is unavailable.
• Adversarial and consistency constraints are used to preserve content.
• CycleGAN is a common unpaired translation model.
• Applications include season transfer, artistic style transfer, and synthetic-to-real adaptation.

Paired training usually provides stronger supervision but aligned datasets can be expensive. Unpaired training is more flexible, although the mapping is less constrained and may introduce unwanted semantic changes.

StyleGAN separates the source of visual style from the direct convolutional synthesis process.

Key ideas:

• A mapping network transforms an input latent vector into an intermediate latent representation .
• The synthesis network begins from a learned constant rather than directly reshaping .
• Learned affine transformations convert into layer-specific style parameters.
• Styles modulate convolutional features at different resolutions, controlling coarse structure, medium-scale attributes, and fine details.
• Independent noise is injected into layers to model stochastic details such as hair strands or skin texture.
• Style mixing applies different latent codes at different layers, encouraging disentanglement.
• Later versions use weight modulation and demodulation to reduce artifacts and improve control.

Because different resolutions influence different visual scales, StyleGAN supports high-fidelity synthesis and meaningful manipulation of generated images.

DCGAN:

• Designed as a convolutional baseline for unconditional image generation.
• Maps random latent vectors to images using convolutional upsampling.
• Requires only a collection of images from one target distribution.
• Provides limited explicit control beyond changes in the latent vector.

CycleGAN:

• Designed for unpaired image-to-image translation.
• Uses two generators, two discriminators, and cycle consistency.
• Requires separate datasets from source and target domains but no aligned pairs.
• Controls output primarily through the supplied source image and chosen target domain.

StyleGAN:

• Designed for high-resolution, high-fidelity image generation and latent manipulation.
• Uses a mapping network, intermediate latent space, and layer-wise style modulation.
• Usually trains on a single visual domain.
• Provides fine-grained control over attributes at different spatial scales.

Thus, DCGAN is a foundational generation architecture, CycleGAN performs domain translation, and StyleGAN emphasizes synthesis quality and controllability.

An image generator learns a nonlinear mapping

where is sampled from a simple prior such as and is a generated image.

During training, the model adjusts its parameters so that the distribution of generated images approximates the training-data distribution. Different objectives achieve this differently: a GAN uses discriminator feedback, while a VAE uses reconstruction and probabilistic regularization.

Latent-space interpolation:

• Choose two latent vectors and .
• Linear interpolation is defined by

• Generate each intermediate image as .
• A well-structured latent space produces gradual semantic changes rather than abrupt or unrealistic transitions.

For spherical priors, spherical interpolation may better follow the high-probability region. Interpolation is useful for inspecting continuity, generating transitions, and studying learned semantic directions.

Single-image super-resolution (SISR) reconstructs a high-resolution image from one low-resolution image . It estimates

The problem is ill-posed because many high-resolution images can produce the same low-resolution observation.

Generative approach:

• A generator upsamples the low-resolution input and predicts missing details.
• A pixel loss such as or encourages fidelity to the ground-truth image.
• A perceptual loss compares deep feature representations and preserves semantic structure.
• An adversarial loss encourages realistic textures by requiring outputs to fool a discriminator.

For example, SRGAN combines content and adversarial losses. Compared with interpolation methods, generative super-resolution can create sharper textures. However, it may hallucinate plausible details that are not present in the original scene, which is risky in medical, scientific, or forensic applications.

A super-resolution generator can be trained with a weighted combination of losses:

Pixel loss:

It promotes correct color, intensity, and global structure. An loss often produces less smoothing than .

Perceptual loss:

where denotes features from a pretrained network. It encourages similarity in semantic content and texture.

Adversarial loss:

It encourages outputs to lie on the manifold of realistic high-resolution images.

The coefficients , , and balance distortion and perceptual realism. Excessive adversarial weight can produce convincing but incorrect details.

CLIP learns a shared embedding space for images and natural-language descriptions.

Architecture:

• An image encoder converts an image into an embedding .
• A text encoder converts a caption into an embedding .
• The embeddings are normalized, and their similarity is commonly measured by cosine similarity:

For a batch of matching image-text pairs, the model forms an image-text similarity matrix. The contrastive objective increases the similarity of matched pairs and decreases the similarity of mismatched pairs.

A symmetric loss is used:

where each component is a cross-entropy loss over temperature-scaled similarities.

This training allows CLIP to associate visual concepts with language and supports zero-shot classification, cross-modal retrieval, and text-guided generation.

Zero-shot classification:

1. Construct text prompts for every candidate class, such as "a photo of a cat".
2. Encode all prompts using the CLIP text encoder.
3. Encode the test image using the CLIP image encoder.
4. Compute cosine similarities between the image embedding and class-text embeddings.
5. Select the class with the highest similarity, often after applying a softmax.

Prompt ensembling can improve performance by averaging embeddings or predictions from multiple templates.

Text-guided image generation:

• A text prompt is encoded by CLIP.
• A generator's latent vector or parameters are optimized so that the generated image embedding aligns with the text embedding.
• Alternatively, a generative model can be trained with CLIP-derived semantic guidance.

Limitations:

• Results depend on prompt wording and pretraining data.
• CLIP may inherit social or dataset biases.
• High semantic similarity does not guarantee exact spatial structure or factual correctness.

Interpretability techniques help users understand why a vision model produced a particular prediction.

Importance:

• Detects reliance on irrelevant backgrounds or artifacts.
• Supports debugging and model improvement.
• Assists bias and fairness analysis.
• Builds confidence in high-stakes domains.
• Helps verify whether predictions use semantically meaningful evidence.

Major categories:

• Gradient-based methods: Use derivatives of an output with respect to pixels or features, as in saliency maps and Grad-CAM.
• Perturbation-based methods: Mask, alter, or remove input regions and measure prediction changes, as in occlusion sensitivity.
• Activation-based methods: Visualize feature maps or neuron activations.
• Surrogate-model methods: Approximate local model behavior with an interpretable model, as in LIME.
• Example-based methods: Explain predictions using influential, similar, or counterfactual examples.

Interpretability does not automatically establish causality. Explanations should therefore be checked for stability, faithfulness, and sensitivity to meaningful input changes.

Grad-CAM produces a class-specific localization map using a convolutional layer's feature maps.

Let be the score for class , and let denote feature map with spatial dimensions .

Step 1: Compute gradients

Step 2: Globally average the gradients to obtain the importance of feature map :

Step 3: Form a weighted combination of the feature maps:

Step 4: Apply ReLU:

ReLU retains regions that positively support class . The resulting low-resolution map is upsampled to the input-image size and overlaid as a heatmap.

Grad-CAM is class-discriminative and does not require changing or retraining the network, but its localization resolution is limited by the selected convolutional layer.

A basic saliency map measures how sensitive a class score is to each input pixel. For an input image and class score , it is computed as

For a color image, channel gradients may be reduced using a maximum, sum, or norm to produce one importance value per pixel.

Strengths:

• Simple to compute with one backward pass.
• Provides pixel-level resolution.
• Can be applied to differentiable neural networks without retraining.
• Helps identify image regions that locally affect a prediction.

Limitations:

• Maps are often noisy and visually difficult to interpret.
• Gradients can saturate and become small even for important features.
• Small input perturbations may substantially change the explanation.
• High sensitivity does not necessarily imply causal importance.
• The method may highlight edges rather than complete semantic objects.

Methods such as SmoothGrad average saliency over noisy inputs, while Integrated Gradients accumulates gradients along a path from a baseline to the input.

Computation:

• A saliency map differentiates a class score directly with respect to input pixels.
• Grad-CAM differentiates the score with respect to convolutional feature maps and uses pooled gradients as channel weights.

Resolution:

• Saliency maps have pixel-level resolution.
• Grad-CAM maps are coarser because convolutional feature maps have lower spatial resolution and must be upsampled.

Class specificity:

• Both can be computed for a selected class.
• Grad-CAM often gives clearer class-discriminative object regions because it combines high-level semantic features.

Interpretation:

• Saliency indicates local sensitivity: how changes in pixels may affect the output.
• Grad-CAM indicates spatial regions whose high-level features positively support a class score.

Limitations:

• Saliency can be noisy and unstable.
• Grad-CAM may overlook small objects or fine details and depends on the chosen layer.

The methods are complementary: Grad-CAM offers semantic localization, while saliency maps provide finer-grained sensitivity information.