Unit 6: Generative NLP and LLMs

I. Orientation

Generative natural language processing models learn a probability distribution over token sequences and use it to produce new text. Modern large language models (LLMs), typically based on the Transformer architecture, generate text autoregressively: given preceding tokens, they predict a probability distribution for the next token and repeat the process until a stopping condition is reached.

• Governing principle: For tokens (x_1,\ldots,x_T), the chain rule factorizes sequence probability as:

P(x₁, x₂, ..., x_T) = ∏ from t=1 to T P(x_t | x₁, ..., x_{t-1})

• (x_t): token generated at position (t).
• (T): sequence length.
• (P(xt\mid x{<t})): conditional next-token probability.

• Core architecture: Decoder-only Transformers use masked self-attention so each position can attend only to earlier tokens.
• Tokenization convention: Text is represented as tokens, often subword units; one token may be a word, punctuation mark, or word fragment.
• Training objective: Next-token prediction commonly minimizes cross-entropy between predicted probabilities and observed tokens.
• Generation cycle: The model produces logits, converts them to probabilities, selects a token using a decoding strategy, appends it to the context, and repeats.
• Central limitation: A language model estimates linguistic likelihood, not factual truth; probable text can therefore be incorrect.

II. Generative Models

A. Generative NLP Models

Generative NLP models create or transform text by modeling the probability of linguistic sequences.

• Autoregressive models: Models such as GPT generate left to right using (P(xt\mid x{<t})); they naturally support continuation, dialogue, and open-ended writing.
• Encoder-decoder models: Models such as T5 encode an input sequence and condition a decoder on that representation, making them suitable for translation and summarization.
• Pretraining: Models learn broad linguistic patterns from large corpora through objectives such as causal language modeling.
• Conditional generation: A prompt, document, dialogue history, or instruction becomes the context that shapes the output distribution.
• Parameters and scale: Learned weights store distributed statistical patterns; increasing parameters and data can improve capability but also raises computational and governance costs.
• Context window: Generation is conditioned only on tokens available within the model’s maximum context length; omitted or truncated information cannot directly guide prediction.
• Limitation: Training data may encode bias, errors, private information, or obsolete facts, all of which can influence generated output.

III. Text Generation Strategies

A. Text Generation Strategies

Text generation strategies convert next-token probability distributions into concrete output sequences.

• Probability conversion: Given logits (z_i), softmax assigns token (i) the probability:

P(i) = exp(z_i / τ) / Σ_j exp(z_j / τ)

• (z_i): model score for token (i).
• (\tau): temperature; lower values sharpen and higher values flatten the distribution.
• (j): index over vocabulary tokens.

• Deterministic decoding: Greedy and standard beam search return the same result for unchanged model states and settings.
• Stochastic decoding: Top-k and nucleus sampling draw randomly from a restricted distribution, allowing multiple valid outputs.
• Quality trade-off: Concentrated selection improves predictability, while broader sampling increases diversity but can introduce incoherence.
• Stopping conditions: Generation ends at an end-of-sequence token, a length limit, or an application-defined stop sequence.

B. Greedy Search

Greedy search chooses the highest-probability token at every generation step.

• Rule: At step (t), select:

x_t = argmax_x P(x | x_<t)

• (x): candidate next token.
• (x_{<t}): all previously generated tokens.

• Advantage: It requires only one active sequence, so decoding is fast and memory-efficient.
• Myopic behavior: The locally best token need not belong to the globally most probable sequence.
• Typical use: It suits constrained, low-latency tasks where consistency matters more than output diversity.
• Failure mode: Repeated high-probability choices can produce generic wording or repetition, especially in open-ended generation.

C. Beam Search

Beam search approximately maximizes whole-sequence probability by retaining several promising partial sequences.

• Beam mechanism: With beam width (B), each step expands candidates and keeps the (B) sequences with the highest cumulative scores.
• Sequence score: Log probabilities are added to avoid numerical underflow:

score(x₁:T) = Σ from t=1 to T log P(x_t | x_<t)

• Length bias: Raw log-probability favors shorter sequences because each added log probability is non-positive; length normalization or penalties can compensate.
• Contrast with greedy search:
  1. Greedy: Keeps one path and cannot recover from an early poor choice.
  2. Beam: Keeps (B>1) paths, increasing search coverage and computational cost.
• Best fit: Beam search is useful when outputs have relatively narrow correct forms, such as translation, but may reduce diversity in dialogue.

D. Top-k Sampling

Top-k sampling randomly selects the next token from the (k) highest-probability candidates.

• Filtering rule: Tokens outside the top (k) receive probability zero; probabilities inside the set are renormalized before sampling.
• Concrete case: If (k=3), only the three most probable tokens can be selected, even when the vocabulary contains 50,000 tokens.
• Control effect: Small (k) produces conservative text; large (k) permits more varied and potentially less coherent output.
• Fixed-size weakness: The same (k) is used whether probability is concentrated among two tokens or spread across hundreds.
• Application: It is commonly combined with temperature for creative generation and conversational variation.

E. Nucleus Sampling

Nucleus sampling, or top-p sampling, draws from the smallest token set whose cumulative probability reaches a threshold (p).

• Dynamic set: Tokens are sorted by probability, then included until their cumulative mass is at least (p), typically with (0<p\leq1).
• Concrete case: For probabilities (0.45, 0.30, 0.15, 0.10) and (p=0.80), the first three tokens form the nucleus because their cumulative probability is (0.90).
• Adaptive behavior: The candidate set shrinks when the model is confident and expands when uncertainty is distributed.
• Contrast with top-k:
  1. Top-k: Maintains a fixed number of candidates.
  2. Nucleus: Maintains a variable number covering a fixed probability mass.
• Limitation: High (p) or high temperature can admit unlikely tokens and weaken factual or logical consistency.

IV. Instruction Following

A. Instruction-tuned Large Language Models

Instruction-tuned large language models are pretrained LLMs further optimized to respond usefully to natural-language directions.

• Supervised fine-tuning: The model learns from prompt-response pairs demonstrating tasks such as classification, extraction, rewriting, and explanation.
• Preference alignment: Human or model preferences can train a reward model, followed by optimization methods such as reinforcement learning from human feedback.
• Direct preference methods: Approaches such as direct preference optimization learn from preferred and rejected responses without a separate reinforcement-learning loop.
• Prompt sensitivity: Output changes with wording, examples, role specifications, ordering, and requested format because all become conditioning tokens.
• Emergent interface: One model can perform many tasks through instructions without task-specific parameter updates.
• Alignment limitation: Helpfulness and harmlessness training does not guarantee truthfulness, robustness, or faithful compliance with every constraint.

V. Task-Specific Model Behaviors

A. Model Behaviors in Summarization

In summarization, generative models compress source content while attempting to preserve its central meaning.

• Extractive tendency: The model may reuse source phrases, especially when factual precision is strongly favored.
• Abstractive tendency: It can paraphrase and synthesize information, producing fluent summaries that are not direct source spans.
• Faithfulness: Every asserted fact should be supported by the source; fabricated names, numbers, or causal links are intrinsic hallucinations.
• Coverage: A summary must retain salient information while omitting repetition and minor detail.
• Length control: Explicit token, sentence, or format constraints influence compression, although exact compliance is not guaranteed.
• Evaluation difficulty: Multiple summaries can be valid, so overlap with one reference may underestimate quality.

B. Dialogue Generation

Dialogue generation produces context-sensitive turns while maintaining conversational relevance and coherence.

• Context tracking: The model uses dialogue history to resolve references, preserve topic, and avoid contradictory replies.
• Role consistency: System instructions and dialogue labels establish expected identity, tone, and behavioral boundaries.
• Response diversity: Sampling avoids identical replies but can cause topic drift or inconsistent claims.
• Safety behavior: Aligned systems may refuse harmful requests, redirect users, or provide constrained information.
• Long-dialogue limitation: Earlier details may be lost when context is truncated or overwhelmed by newer tokens.
• Anthropomorphism risk: First-person fluency can suggest beliefs or emotions, although generation remains probabilistic computation.

C. Reasoning Tasks

On reasoning tasks, LLMs generate intermediate language patterns intended to connect premises with conclusions.

• Task forms: Arithmetic, logical deduction, planning, and multi-step question answering require dependencies across several generated steps.
• Prompted decomposition: Requesting intermediate stages can improve performance by turning one difficult prediction into multiple smaller predictions.
• Tool support: Calculators, code interpreters, search systems, and retrieval databases can verify operations or supply current evidence.
• Fragility: A plausible intermediate step may be invalid, and later steps can amplify the error.
• Consistency methods: Sampling several reasoning paths and aggregating final answers can improve reliability when correct paths recur.
• Key distinction: A correct answer does not prove that the generated explanation faithfully represents the model’s internal computation.

VI. Evaluation

A. Evaluation Metrics

Evaluation metrics measure generation quality, but no single metric captures fluency, correctness, usefulness, safety, and diversity.

• Reference-based metrics: BLEU and ROUGE measure n-gram overlap; they are reproducible but penalize valid paraphrases.
• Semantic metrics: Embedding-based measures compare contextual representations and better recognize similar meanings.
• Task metrics: Exact match, accuracy, factual consistency, or execution success may be appropriate when outputs have verifiable targets.
• Diversity metrics: Distinct-n measures the proportion of unique n-grams, though diversity alone does not imply quality.
• Benchmark risk: Data contamination, narrow prompts, and metric optimization can inflate apparent capability.
• Best practice: Evaluation should combine automatic metrics, task-specific checks, and human assessment.

B. Perplexity

Perplexity measures how surprised a language model is by a token sequence.

• Definition: For (N) observed tokens, perplexity is:

PPL = exp[-(1/N) Σ from t=1 to N log P(x_t | x_<t)]

• (N): number of evaluated tokens.
• (x_t): observed token at position (t).
• Lower perplexity indicates greater assigned probability.

• Interpretation: A perplexity of 20 can be viewed loosely as uncertainty comparable to choosing among 20 equally likely options per step.
• Comparability condition: Models should use the same dataset and compatible tokenization; token-level values across different vocabularies may be misleading.
• Limitation: Low perplexity measures predictive fit, not factual accuracy, safety, reasoning validity, or user satisfaction.

C. Human Judgment Measures

Human judgment measures assess qualities that automatic scores cannot adequately represent.

• Rating scales: Evaluators score fluency, relevance, coherence, factuality, helpfulness, or safety using defined rubrics.
• Pairwise preference: Judges choose the better of two outputs; this is often more reliable than assigning absolute scores.
• Agreement: Statistics such as Cohen’s kappa assess consistency beyond chance between annotators.
• Bias controls: Randomized order, blinded model identity, diverse evaluators, and clear criteria reduce presentation and demographic bias.
• Limitation: Human evaluation is costly, slow, culturally dependent, and sensitive to evaluator expertise.
• Reliable design: Report sample size, rubric, evaluator population, and agreement rather than only an average score.

VII. Reliability and Interpretation

A. Explainability in LLMs

Explainability in LLMs seeks to clarify why a model produced an output or how internal representations contribute to behavior.

• Attention analysis: Attention weights show token-to-token information routing, but they are not automatically causal explanations.
• Feature attribution: Gradient-based or perturbation methods estimate how changing input tokens affects an output score.
• Probing: Classifiers test whether internal activations encode properties such as syntax, entities, or sentiment.
• Mechanistic interpretability: Researchers inspect neurons, features, and computational circuits to identify causal internal mechanisms.
• Generated explanations: Natural-language rationales are readable but may be post-hoc narratives rather than faithful internal accounts.
• Evaluation criterion: An explanation should be understandable and causally faithful; plausibility alone is insufficient.

B. Hallucination in LLMs

Hallucination is the generation of unsupported, false, or internally inconsistent content presented as if it were valid.

• Intrinsic hallucination: The output contradicts the supplied source, such as changing a document’s stated date.
• Extrinsic hallucination: The output introduces a claim that cannot be verified from the provided source or trusted evidence.
• Causes: Next-token optimization rewards probable continuation; incomplete context, ambiguous prompts, noisy training data, and aggressive sampling increase risk.
• Detection: Claims can be checked against source passages, structured databases, external tools, or consistency tests.
• Mitigation: Retrieval-augmented generation supplies evidence; constrained decoding, citations, tool use, and post-generation verification narrow unsupported output.
• Calibration: Systems should communicate evidential limits and abstain when reliable support is unavailable.
• Residual risk: Retrieval and citations do not guarantee correctness because evidence may be irrelevant, outdated, misread, or inaccurately attributed.