Unit 5: Transformers and Pretrained Language Models

I. Orientation

Transformers are neural architectures introduced for sequence modelling through attention rather than recurrence. Their central principle is to represent each token using relationships with other tokens, allowing efficient parallel processing and effective transfer from large-scale pretraining to downstream NLP tasks.

• Governing principle: Contextual representations are computed from weighted interactions among token representations.
• Core convention: A sequence is converted into token IDs, embedded as vectors, enriched with position information, and processed through stacked layers.
• Training distinction: Encoder models usually learn bidirectional context; decoder models predict tokens from preceding context; encoder-decoder models transform one sequence into another.
• Transfer assumption: General linguistic patterns learned from large corpora can be adapted to smaller supervised datasets.
• Common notation: (d_{\text{model}}) denotes hidden-vector size, (L) the sequence length, (h) the number of attention heads, and (d_k) the key/query dimension.

II. Transformer architecture — attention-based sequence modelling

A. Transformer architecture

The Transformer architecture maps an input sequence to contextual representations using embeddings, attention, feed-forward networks, residual connections, and normalization.

• Input representation: Each token ID becomes a vector in (\mathbb{R}^{d_{\text{model}}}); positional information is added before the first layer.
• Layer composition: A typical layer contains multi-head attention followed by a position-wise feed-forward network.
• Parallelism: Unlike an RNN, all tokens in a training sequence can be processed simultaneously, improving hardware utilization.
• Complexity: Full self-attention requires (O(L^2d_{\text{model}})) pairwise interaction work, which becomes expensive for long documents.

III. Self-attention — contextual token interaction

A. Self-attention

Self-attention computes a representation for each token by weighting every token, including itself, according to learned relevance.

• Query, key, value: The input matrix (X) is projected into (Q=XW_Q), (K=XW_K), and (V=XW_V).
• Scaled attention: The standard operation is:

Attention(Q, K, V) = softmax(QK^T / sqrt(d_k)) V

Here, (QK^T) contains token-to-token similarity scores, (d_k) prevents large dot products from producing overly sharp softmax values, and (V) supplies the information combined into outputs.

• Context dependence: In “The animal did not cross,” the representation of “animal” can incorporate information from “did not cross.”
• Masking: Padding masks exclude absent tokens; causal masks prevent a decoder position from seeing future tokens.

IV. Multi-head attention — several relation spaces

A. Multi-head attention

Multi-head attention runs several attention operations in parallel so the model can capture different relationships among the same tokens.

• Separate projections: Head (i) uses (W_Q^{(i)}, W_K^{(i)}, W_V^{(i)}), allowing one head to focus on syntax while another tracks long-distance references.
• Concatenation: Head outputs are concatenated and projected:

MultiHead(X) = Concat(head_1, ..., head_h) W_O

(h) is the number of heads and (WO) maps the concatenated result back to (d{\text{model}}).

• Dimension example: With (d_{\text{model}}=512) and (h=8), each head commonly uses (d_k=64), since (8 \times 64=512).
• Benefit: Multiple heads provide distinct learned views without requiring a single attention distribution to represent every relation.

V. Positional encoding — representing order

A. Positional encoding

Positional encoding supplies sequence-order information because self-attention alone is permutation-invariant.

• Sinusoidal form: For position (p) and dimension (2i):

PE(p, 2i)   = sin(p / 10000^(2i / d_model))
PE(p, 2i+1) = cos(p / 10000^(2i / d_model))

(p) is the token position, (i) indexes dimensions, and (d_{\text{model}}) is the embedding width.

• Addition: The positional vector is added to the token embedding, preserving the fixed hidden size.
• Relative information: Sinusoidal patterns allow the model to derive useful relationships between positions, while learned position embeddings directly optimize a vector for each permitted position.
• Limitation: Learned absolute positions generally do not extrapolate reliably beyond the maximum training length.

VI. Transformer encoder and decoder blocks — reusable processing units

A. Transformer encoder and decoder blocks

Encoder and decoder blocks differ mainly in their attention patterns and whether they condition on another sequence.

• Encoder block: It contains bidirectional self-attention, a feed-forward network, residual connections, and layer normalization.
• Decoder block: It contains masked self-attention, encoder-decoder cross-attention when applicable, and a feed-forward network.
• Feed-forward network: Each position is transformed independently:

FFN(x) = max(0, xW_1 + b_1)W_2 + b_2

(W_1,W_2) are learned matrices and (b_1,b_2) biases; modern variants may replace ReLU with GELU.

• Residual path: Adding a sublayer output to its input, followed by normalization, supports gradient flow through deep stacks.

VII. Tokenization methods — converting text into model units

A. Tokenization methods

Tokenization divides raw text into units that can be mapped to a finite vocabulary while balancing vocabulary size and sequence length.

• Whitespace tokenization: “Transformers process text” becomes three word tokens, but unseen words and spelling variants cause vocabulary problems.
• Character tokenization: It handles arbitrary words but produces longer sequences and may make semantic learning harder.
• Subword tokenization: Frequent words remain whole while rare words are decomposed, such as “play” plus “##ing.”
• Special tokens: Models may reserve [CLS], [SEP], [MASK], or <pad> for classification, separation, masking, and batch alignment.

VIII. Byte-Pair Encoding — frequency-based subword construction

A. Byte-Pair Encoding

Byte-Pair Encoding (BPE) builds a vocabulary by repeatedly merging the most frequent adjacent symbol pair in a training corpus.

• Initialization: A word such as “lower” can begin as l o w e r or as byte-level symbols, depending on implementation.
• Merge rule: If l and o frequently occur together, the pair becomes lo; later merges may create low.
• Encoding: New text applies learned merges in priority order, so an unseen word can still be represented through known subwords.
• Trade-off: More merges shorten sequences but enlarge vocabulary; byte-level BPE, used by GPT-2-style systems, can represent any Unicode text through byte units.

IX. WordPiece — likelihood-oriented subwords

A. WordPiece

WordPiece selects subword units using a vocabulary-building objective related to improving corpus likelihood rather than simply maximizing pair frequency.

• Boundary marking: BERT commonly writes continuation pieces with ##; “playing” may become play and ##ing.
• Unknown handling: A word that cannot be decomposed is mapped to [UNK], unlike byte-level schemes that can usually encode every input.
• Greedy encoding: The tokenizer typically selects the longest valid vocabulary piece at each position.
• Practical effect: WordPiece reduces out-of-vocabulary words while preserving common morphemes and keeping sequence lengths manageable.

X. Pretrained transformer models — learning before adaptation

A. Pretrained transformer models

Pretrained transformer models learn general language structure from large unlabelled or weakly labelled corpora before being adapted to specific tasks.

• Pretraining objective: The loss may predict masked tokens, next tokens, or a target sequence conditioned on an input sequence.
• Representation transfer: Early layers often encode local and syntactic patterns, while later layers become more task- and context-sensitive.
• Scale factors: Performance depends on architecture, corpus quality, token vocabulary, parameter count, and compute.
• Limitation: Pretraining data can encode social bias, factual errors, privacy risks, or domain mismatch.

XI. BERT — bidirectional encoder pretraining

A. BERT

BERT is an encoder-only Transformer designed to learn bidirectional contextual representations for language understanding tasks.

• Input format: A pair may be represented as [CLS] sentence A [SEP] sentence B [SEP]; the [CLS] vector is commonly used for classification.
• Architecture: BERT-Base uses 12 encoder layers, hidden size 768, and 12 attention heads.
• Strength: Every token can attend to tokens on both sides, supporting classification, tagging, and extractive question answering.
• Limitation: It is not naturally a left-to-right text generator because its pretraining context is bidirectional.

XII. GPT — autoregressive decoder pretraining

A. GPT

GPT models use decoder-style causal self-attention to predict the next token from preceding tokens.

• Objective: For tokens (x_1,\ldots,x_T), training maximizes:

P(x_1, ..., x_T) = product from t=1 to T of P(x_t | x_1, ..., x_(t-1))

• Generation: At inference, a predicted token is appended to the context and the model repeats the process.
• Strength: The same next-token mechanism supports completion, dialogue, summarization, and code generation.
• Limitation: Errors can accumulate during long generation, and fluent output is not a guarantee of factual accuracy.

XIII. T5 — text-to-text transfer

A. T5

T5 frames nearly every NLP problem as generating target text from input text using an encoder-decoder Transformer.

• Unified format: Classification can use sentiment: This film is excellent with target positive.
• Pretraining: T5 corrupts spans of text and trains the decoder to reconstruct them, using sentinel tokens to mark missing spans.
• Task flexibility: Translation, summarization, question answering, and classification share the same input-output interface.
• Cost: Encoder-decoder generation can require more computation than producing a single encoder classification vector.

XIV. Masked language modeling — reconstructing hidden tokens

A. Masked language modeling

Masked language modeling trains an encoder to predict selected tokens from their surrounding context.

• Corruption pattern: BERT selects roughly 15% of positions; selected tokens may be replaced by [MASK], left unchanged, or replaced randomly.
• Loss: Cross-entropy is computed only at selected positions:

L_MLM = - sum over i in M of log P(x_i | x_not_in_M)

(M) is the set of masked positions and (x_i) the original token.

• Benefit: The model learns both left and right context, useful for language understanding.
• Mismatch: [MASK] appears during pretraining but usually not in ordinary downstream text.

XV. Next sentence prediction — sentence-pair discrimination

A. Next sentence prediction

Next sentence prediction trains BERT to decide whether a second segment follows the first in the original corpus.

• Labels: A positive pair is consecutive; a negative pair combines segments sampled from different locations.
• Classifier input: The [CLS] representation feeds a binary classification layer.
• Purpose: The task was intended to teach inter-sentence relationships, complementing token-level masking.
• Qualification: Later research found that removing NSP or replacing it with related sentence-order objectives can improve some model variants.

XVI. Causal language modeling — left-to-right prediction

A. Causal language modeling

Causal language modeling predicts each token using only earlier tokens, enforced by a triangular attention mask.

• Mask structure: Position (t) may attend to positions (1) through (t), but not (t+1) through (T).
• Training signal: Every next-token prediction contributes to the loss, unlike masked language modeling, which supervises selected positions.
• Use: The objective directly matches autoregressive generation, including GPT-style completion.
• Sampling: Temperature modifies logits before softmax; lower values make outputs more deterministic, while higher values increase variation.

XVII. Transfer learning for NLP tasks — adapting general knowledge

A. Transfer learning for NLP tasks

Transfer learning reuses a pretrained model and adapts its representations to a task with labelled examples.

• Workflow: Tokenize task data, load pretrained weights, attach a task head, optimize a supervised loss, and evaluate on held-out data.
• Advantages: It reduces labelled-data requirements and typically converges faster than training from random initialization.
• Adaptation choices: Full fine-tuning updates all weights; parameter-efficient methods update small adapter, prompt, or low-rank components.
• Risk: Excessive training on a small dataset can overfit or cause catastrophic forgetting of general capabilities.

XVIII. Fine-tuning for text classification — supervised prediction

A. Fine-tuning for text classification

Fine-tuning for text classification adds a prediction layer to a pretrained representation and trains it using labelled document or sentence categories.

• Input and head: BERT-style systems feed the [CLS] vector (h_{\text{CLS}}) to:

p = softmax(W h_CLS + b)

(W) and (b) are task-head parameters and (p) contains class probabilities.

• Binary tasks: Sentiment labels such as positive or negative commonly use two output logits and cross-entropy.
• Multilabel tasks: Independent sigmoid outputs are used when one text can receive several labels.
• Practical control: Small learning rates, validation monitoring, and class-balanced metrics help prevent overfitting.

XIX. Named entity recognition — token-level labelling

A. Named entity recognition

Named entity recognition (NER) assigns labels such as PER, ORG, and LOC to spans identifying people, organizations, and locations.

• BIO scheme: B-ORG begins an organization, I-ORG continues it, and O marks a non-entity token.
• Model output: A contextual vector at each token position produces a label distribution, often with a linear classifier or a CRF layer.
• Subword alignment: Labels may be assigned to the first subword and ignored or copied for continuation pieces, depending on the dataset convention.
• Evaluation: Entity-level precision, recall, and F1 require correctly matching the complete span and type, not merely individual tokens.

XX. Question answering — locating or generating answers

A. Question answering

Question answering adapts Transformers to produce an answer from a question and supporting context.

• Extractive approach: BERT predicts start and end positions in a passage; the span “Paris” might correspond to start index 5 and end index 5.
• Output scores: Separate vectors estimate (P{\text{start}}(i)) and (P{\text{end}}(j)), with invalid spans such as (j<i) rejected.
• Generative approach: T5 or similar encoder-decoder models generate an answer sequence rather than selecting text positions.
• Evaluation: Exact Match requires string agreement; token-level F1 gives partial credit for overlapping answer words.

XXI. HuggingFace Transformers — implementation ecosystem

A. HuggingFace Transformers

HuggingFace Transformers provides pretrained model classes, tokenizers, configuration objects, datasets integrations, and training utilities.

• Loading components:

from transformers import AutoTokenizer, AutoModelForSequenceClassification

tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")
model = AutoModelForSequenceClassification.from_pretrained(
    "bert-base-uncased", num_labels=2
)

The tokenizer converts text into model inputs; the model adds a two-class classification head.

• Batch encoding: tokenizer(texts, padding=True, truncation=True, return_tensors="pt") creates padded tensors and attention masks.
• Task classes: AutoModelForTokenClassification supports NER, while AutoModelForQuestionAnswering predicts answer spans.
• Training interface: Trainer can manage batching, evaluation, logging, checkpoints, and optimization, but correct labels, truncation strategy, and metric computation remain the practitioner’s responsibility.