Unit 6: Generative Adversarial Networks

I. Orientation — Adversarial Learning as Generative Modeling

Generative Adversarial Networks (GANs), introduced by Ian Goodfellow and collaborators in 2014, learn to generate data through competition between two neural networks. A generator produces synthetic samples, while a discriminator attempts to distinguish generated samples from real training data.

• Governing principle: Training is formulated as a two-player minimax game in which each network improves by opposing the other.
• Data distribution: Real observations follow an unknown distribution (p_{\text{data}}(x)), approximated using a finite training dataset.
• Latent distribution: The generator receives a random vector (z\sim p_z(z)), commonly sampled from a standard normal or uniform distribution.
• Implicit modeling: A GAN generates samples without explicitly calculating the probability density (p(x)).
• Differentiable components: Both generator and discriminator are usually neural networks trained through backpropagation.
• Equilibrium objective: Ideally, the generated distribution (pg) becomes equal to (p{\text{data}}), making generated and real samples indistinguishable.
• Major extensions: Conditional GANs control generation through labels, while CycleGAN performs translation using unpaired datasets.
• Operational context: Docker packages models and dependencies, while Streamlit provides a web interface for deployment on GPU-enabled NVIDIA servers.

II. Generative Models — Learning a Data Distribution

A. Introduction to generative models

A generative model learns the structure of observed data so that it can create new samples resembling the training examples.

• Objective: Given examples (x_1,\ldots,xn), estimate or approximate (p{\text{data}}(x)) and sample new observations from it.
• Generative versus discriminative:
  1. Generative model: Learns (p(x)) or the joint distribution (p(x,y)); examples include GANs, variational autoencoders and diffusion models.
  2. Discriminative model: Learns (p(y\mid x)) or a decision boundary; examples include image classifiers and logistic regression.
• Explicit models: Autoregressive models and normalizing flows define a tractable or approximated likelihood (p_\theta(x)).
• Implicit models: GANs define a sampling process (x=G_\theta(z)) without requiring an explicit density value.
• Applications: Concrete uses include image synthesis, super-resolution, data augmentation, inpainting, style transfer and synthetic medical data.
• Evaluation: Generated samples are assessed through visual quality, diversity and metrics such as Fréchet Inception Distance; lower FID generally indicates closer real and generated feature distributions.
• Limitation: A model may produce realistic individual images while omitting important regions of the data distribution.

III. GAN Architecture — Competing Neural Networks

A. Overview of GAN structure

A GAN connects a generator and discriminator so that their opposing objectives create a useful learning signal.

• Data flow:
  • Sample real data (x\sim p_{\text{data}}).
  • Sample latent noise (z\sim p_z).
  • Produce a synthetic sample (\hat{x}=G(z)).
  • Ask (D) to classify (x) as real and (\hat{x}) as fake.
• Minimax objective:

min_G max_D V(D,G)
= E_x[log D(x)] + E_z[log(1 - D(G(z)))]

Here, (G) is the generator, (D(x)\in(0,1)) is the discriminator’s estimated probability that (x) is real, and (E) denotes expectation.

• Alternating optimization: (D) is updated using real and fake batches; (G) is then updated while discriminator parameters are held fixed.
• Ideal equilibrium: When (pg=p{\text{data}}), an optimal discriminator returns (D(x)=0.5), because neither source is more likely.
• Common architecture: DCGAN uses convolutional layers, batch normalization and learned upsampling for image generation.

B. Discriminator

The discriminator is a binary classifier that supplies the generator with feedback about the realism of synthetic samples.

• Input and output: It receives an image (x) and returns a scalar (D(x)), often produced by a sigmoid output layer.
• Training targets: Real samples normally use label (1), while (G(z)) samples use label (0).
• Loss function:

L_D = -[E_x log D(x) + E_z log(1 - D(G(z)))]

Minimizing (L_D) is equivalent to maximizing the discriminator part of the minimax objective.

• Architecture: Image discriminators commonly use strided convolutions and Leaky ReLU activations before a scalar output.
• Balance requirement: An excessively accurate discriminator can return near-zero probabilities for generated images, leaving the generator with weak or unstable gradients.
• Regularization: Label smoothing, spectral normalization, dropout and gradient penalties can limit overconfidence.

C. Generator

The generator transforms a simple latent vector into a structured synthetic sample of the required dimensions.

• Mapping: (G:\mathbb{R}^{d_z}\rightarrow\mathbb{R}^{d_x}), where (d_z) is latent dimension and (d_x) is data dimension.
• Image generation: A vector such as (z\in\mathbb{R}^{100}) may pass through dense and transposed-convolution layers to create a (64\times64\times3) RGB image.
• Non-saturating loss:

L_G = -E_z[log D(G(z))]

This practical alternative asks (G) to maximize the probability that generated samples are classified as real.

• Gradient path: During a generator update, error is propagated through (D(G(z))), but only (G)’s parameters are changed.
• Output activation: A tanh layer is common when image pixels are normalized to ([-1,1]); a sigmoid suits ([0,1]).
• Latent interpolation: Smoothly moving between (z_1) and (z_2) can produce gradual changes in generated attributes.

IV. GAN Construction — Training Workflow

A. Building GAN

Building a GAN requires compatible networks, carefully normalized data and an alternating training loop.

• Preparation: Resize images consistently, normalize pixel values, create shuffled mini-batches and select a latent dimension.
• Model definition: Construct (G), construct (D), and connect them as (z\rightarrow G(z)\rightarrow D(G(z))).
• Update sequence:

for each batch:
    z = sample_noise(batch_size)
    fake = G(z)
    update D using real images labeled 1 and fake images labeled 0

    z = sample_noise(batch_size)
    freeze D parameters
    update G through D(G(z)) using target label 1

• Optimizers: Adam is frequently used; DCGAN implementations commonly start near a learning rate of (0.0002), subject to tuning.
• Monitoring: Save generated image grids and checkpoints periodically; losses alone do not reliably indicate image quality.
• Implementation caution: “Freezing” (D) for the combined update must not prevent its independent discriminator update.
• Validation: Compare sample diversity, nearest training images and FID to detect memorization or limited coverage.

V. GAN Training Challenges — Instability and Failure Modes

A. Problems with GANs

GANs are difficult to train because two changing models create a non-stationary optimization problem rather than a single fixed objective.

• Mode collapse: Different latent vectors produce nearly identical outputs, so (G) represents only a few modes of the real distribution.
• Vanishing gradients: If (D) rejects fake samples with extreme confidence, the original minimax generator loss can provide little useful gradient.
• Oscillation: Improvements by one network continually alter the other network’s objective, preventing convergence.
• Imbalance: A powerful discriminator overwhelms the generator, whereas a weak discriminator provides inaccurate guidance.
• Evaluation difficulty: Low loss does not guarantee realistic or diverse samples; visual inspection alone is subjective.
• Artifacts: Transposed convolutions may introduce checkerboard patterns because of uneven kernel overlap.
• Mitigation: Non-saturating loss, Wasserstein objectives, gradient penalty, spectral normalization, balanced update ratios and careful initialization improve stability.
• Data sensitivity: Small, biased or heterogeneous datasets can cause memorization, poor coverage and biased generated content.

VI. Unpaired Image Translation — Cycle-Consistent Learning

A. CycleGAN

CycleGAN translates images between two visual domains without requiring paired examples of the same scene.

• Domains: Let (X) represent one domain, such as horses, and (Y) another, such as zebras.
• Components: Two generators, (G:X\rightarrow Y) and (F:Y\rightarrow X), work with discriminators (D_Y) and (D_X).
• Adversarial losses: (D_Y) distinguishes real (Y) images from (G(x)), while (D_X) distinguishes real (X) images from (F(y)).
• Cycle consistency:

L_cyc = E_x ||F(G(x)) - x||_1 + E_y ||G(F(y)) - y||_1

The (L_1) norm penalizes failure to reconstruct an input after a forward-and-backward translation.

• Total objective: Adversarial losses are combined with (\lambda L_{\text{cyc}}), where (\lambda) controls reconstruction importance.
• Identity loss: Passing an image already belonging to the target domain through the relevant generator can discourage unnecessary color changes.
• Limitation: Cycle consistency does not guarantee semantically correct translation, especially when the domains differ greatly in geometry.

VII. Adversarial Robustness — Input-Space Attacks

A. Adversarial FGSM

The Fast Gradient Sign Method creates an adversarial example by moving an input in the direction that increases a trained model’s loss.

• Formula:

x_adv = clip(x + ε sign(∇_x J(θ, x, y)))

Here, (x) is the original input, (x_{\text{adv}}) is the perturbed input, (\epsilon) limits perturbation magnitude, (J) is loss, (\theta) denotes model parameters, and (y) is the true label.

• Mechanism: FGSM performs one gradient step with respect to input pixels rather than model weights.
• Constraint: Under an (L_\infty) bound, every pixel changes by at most (\epsilon), followed by clipping to the valid pixel range.
• Purpose: It tests model robustness and can generate examples for adversarial training.
• Contrast with GANs: GAN “adversarial” training involves competing networks; FGSM involves a deliberately perturbed input attacking one predictive model.
• Limitation: A large (\epsilon) makes perturbations visible, while a small value may fail against robust models.

VIII. Containerized Execution — Reproducible Environments

A. Use of Docker

Docker packages the application, model code and software dependencies into a reproducible container image.

• Image and container: An image is an immutable template; a container is a running instance of that image.
• Dockerfile:

FROM nvcr.io/nvidia/pytorch:24.01-py3
WORKDIR /app
COPY . /app
RUN pip install --no-cache-dir streamlit
EXPOSE 8501
CMD ["streamlit", "run", "app.py", "--server.address=0.0.0.0"]

• GPU access: NVIDIA Container Toolkit allows containers to use the host GPU and compatible NVIDIA driver.
• Build and run:

docker build -t gan-app .
docker run --gpus all -p 8501:8501 gan-app

• Benefits: Versioned environments reduce dependency conflicts and support consistent testing, transfer and rollback.
• Security: Avoid embedding credentials, use minimal images and run containers without unnecessary privileges.

IX. GPU Web Deployment — Interactive Model Serving

A. Model deployment on NVIDIA Server using Streamlit framework

Streamlit can expose a trained GAN through a browser interface while inference runs on an NVIDIA GPU server.

• Model loading: Load the saved checkpoint once with @st.cache_resource, select cuda when available, and call model.eval().
• Inference pattern:

device = "cuda" if torch.cuda.is_available() else "cpu"
model = load_generator().to(device).eval()

with torch.no_grad():
    z = torch.randn(1, latent_dim, device=device)
    image = model(z)

• Interface: st.button, st.slider and st.image can collect generation settings and display converted output.
• Server binding: Run Streamlit with address 0.0.0.0 and map port 8501 through Docker or the server firewall.
• GPU verification: nvidia-smi confirms GPU visibility, memory consumption and active processes.
• Production controls: Validate inputs, limit concurrency, log failures, protect the endpoint with authentication and place a reverse proxy such as Nginx in front of Streamlit.
• Performance: Use torch.no_grad(), mixed precision where validated, cached weights and bounded batch sizes to reduce latency and GPU memory use.