Unit 1: Foundations of Generative AI

Generative AI (emerging as a distinct field from roughly 2014 with Generative Adversarial Networks, and reaching public prominence from 2022 with large language models) refers to machine-learning systems that produce new content—text, images, audio, code—rather than merely classifying or predicting labels on existing data. This unit establishes what "generative" means, the mechanics that make generation possible, the principal model families, and the domains where they are deployed.

Defining properties this unit relies on:

• Generative vs. discriminative: A discriminative model learns P(y | x)—the probability of a label given an input. A generative model learns P(x) or P(x | y)—the distribution of the data itself—so it can sample new instances x from that distribution.
• Learned distribution: The system approximates the probability distribution of a training corpus, then draws novel samples from that approximation.
• Latent representation: Content is encoded into a compressed vector space (the latent space); generation is decoding a point in that space back into a full artefact.
• Probabilistic sampling: Output is stochastic—the same prompt can yield different results because tokens or pixels are sampled from a distribution, not deterministically retrieved.
• Self-supervision: Most training uses unlabelled data, deriving the target from the input itself (e.g., predicting a masked or next word).

II. Defining Generative AI

What makes a model "generative"

Generative AI is the class of models that estimate a data distribution and sample from it to create artefacts statistically consistent with, but not identical to, their training data.

A. Define generative AI

The defining act is generation: producing plausible new data points rather than decisions about existing ones.

• Formal goal: Learn a model distribution p_θ(x) parameterised by weights θ that approximates the true data distribution p_data(x). Sampling x_new ~ p_θ(x) yields new content.
  • θ: the trainable parameters (weights) of the network.
  • x: a data instance (a sentence, image, audio clip).
• Novelty, not retrieval: A generated image of "a cat astronaut" is synthesised pixel-by-pixel; it was never in the dataset, yet is consistent with learned concepts of cat and astronaut.
• Content modalities: Text, images, audio, video, 3D structures, molecular structures, and source code.
• Contrast with discriminative AI:
  1. Discriminative: spam filter answers "is this email spam?"—outputs a label.
  2. Generative: answers "write me an email"—outputs new text drawn from p_θ(x).
• Foundation-model framing: Modern generative systems are often foundation models—large models pre-trained on broad data, then adapted to many tasks.

III. How Generative AI Works

From training data to sampled output

Generation is a two-phase process: a training phase that fits p_θ(x) to data, and an inference phase that samples from it.

A. Explain how generative AI works

The mechanism is: encode data into a numeric space, learn its statistical structure, then decode samples back into content.

• Tokenisation / encoding: Raw input is split into units—tokens for text, patches for images—and mapped to embeddings (dense vectors) so relationships become geometric distances.
  • Example: "king" − "man" + "woman" ≈ "queen" in embedding space.
• The training objective: Adjust θ to raise the likelihood of the training data. For autoregressive text models this is next-token prediction:

  Loss = − Σ_t log p_θ(x_t | x_1, x_2, ..., x_{t-1})

• x_t: the token at position t.
• x_1 … x_{t-1}: all preceding tokens (the context).
• Minimising this cross-entropy loss teaches the model which token most plausibly follows.
  • Optimisation: Gradients of the loss w.r.t. θ are computed by backpropagation; weights are updated by gradient descent over billions of tokens.
  • Inference by sampling: The trained model outputs a probability over the next token; a decoding strategy selects one:
    1. Greedy / deterministic: always take the highest-probability token—safe but repetitive.
    2. Temperature / top-k / top-p sampling: draw stochastically. Temperature T rescales logits (softmax(z/T)); higher T → more diverse, riskier output.
  • Autoregressive loop: Each sampled token is appended to the context and fed back in, generating content one unit at a time until a stop condition.
  • The attention mechanism (transformers): Each token computes weighted relevance to every other token, letting the model condition on long-range context.

  Attention(Q, K, V) = softmax( (Q Kᵀ) / √d_k ) V

• Q, K, V: query, key, value matrices projected from the embeddings.
• d_k: key dimension; √d_k scales dot-products to stabilise gradients.

IV. Generative AI Model Types

The principal architectural families

Different families estimate p_θ(x) by different mechanisms, trading off sample quality, diversity, and controllability.

A. Describe generative AI model types

Each type differs in how it maps between latent space and data space.

• Generative Adversarial Networks (GANs): Two networks compete.
  1. Generator G: maps random noise z to a fake sample G(z).
  2. Discriminator D: scores whether a sample is real or generated.
     • Objective is a minimax game: min_G max_D E[log D(x)] + E[log(1 − D(G(z)))].
     • Strength: sharp images. Weakness: unstable training, mode collapse (limited variety).
• Variational Autoencoders (VAEs): An encoder compresses x to a latent distribution, a decoder reconstructs it.
  • Trained to maximise the evidence lower bound (ELBO): reconstruction accuracy minus a KL-divergence term that keeps the latent space smooth and sampleable.
  • Strength: structured latent space, stable training. Weakness: blurrier outputs than GANs.
• Diffusion models: Learn to reverse a gradual noising process.
  • Forward process: add Gaussian noise to data over many steps until it is pure noise.
  • Reverse process: a network learns to denoise step by step, turning noise into a coherent sample.
  • Powers Stable Diffusion, DALL·E; strength: high-fidelity, diverse images. Weakness: slow multi-step sampling.
• Transformer-based / autoregressive models: Predict the next token using self-attention; the basis of large language models (GPT family).
  • Strength: coherent long-form text and code, scales with data and parameters. Weakness: compute-hungry, can hallucinate.
• Autoregressive vs. one-shot generation:
  1. Autoregressive (transformers): build output sequentially, each step conditioned on the last.
  2. Latent-sampling (GANs/VAEs): produce the whole artefact from one latent vector in a single pass.

V. Generative AI Applications

Where generative models are deployed

Applications map each modality of generation onto a practical task, using the model families above.

A. Describe generative AI applications

The common pattern is: a user supplies a prompt or condition, and the model generates content fulfilling it.

• Text generation and dialogue: Chat assistants, summarisation, translation, drafting—powered by autoregressive transformers.
  • Example: summarising a 20-page report into five bullet points on request.
• Image synthesis and editing: Text-to-image creation, inpainting, style transfer—driven by diffusion and GAN models.
  • Example: prompt "watercolour lighthouse at dusk" → a novel rendered image.
• Code generation: Autocompletion, function synthesis from comments, test writing—transformer models trained on source repositories.
  • Example: a docstring # reverse a string yields a working function body.
• Audio and music: Text-to-speech, voice cloning, music composition from a style prompt.
• Video and 3D: Text-to-video clips and 3D asset generation for games and simulation.
• Scientific and industrial use:
  1. Drug discovery: generating candidate molecular structures with target properties.
  2. Synthetic data: producing artificial datasets to train other models where real data is scarce or private.
• Data augmentation and design: Generating variations to enlarge training sets, or product/architecture design options.
• Limitations shaping deployment:
  • Hallucination: fluent but factually wrong output, because the model optimises plausibility, not truth.
  • Bias: the learned distribution reflects biases in the training corpus.
  • Provenance and misuse: deepfakes, plagiarism risk, and difficulty verifying machine-generated content.
  • Cost: large models require substantial compute for both training and inference.

VI. From Foundations to Practice

How the pieces connect

The unit's four strands form one pipeline, and understanding their linkage is the point of these foundations.

• Definition → mechanism: Because generative AI models P(x) (Section II), it must sample from a learned distribution rather than look content up (Section III).
• Mechanism → model type: The choice of how to represent and sample P(x)—adversarial, variational, diffusion, or autoregressive—defines the family (Section IV).
• Model type → application: The modality a family excels at fixes its natural use: transformers for text and code, diffusion for images, VAEs for structured latent design, GANs for sharp synthetic imagery (Section V).
• Shared vocabulary: Embedding, latent space, sampling, and prompt recur across every section—mastering these four terms is the key to reading any generative-AI system as a variation on one idea: learn a distribution, then draw from it.