Unit 5: Transformers and Pretrained Language Models - Subjective Questions

A Transformer is a neural network architecture that processes sequences using attention mechanisms instead of recurrence or convolution.

Its main components are:

• Input embeddings: Convert tokens into dense vectors.
• Positional encodings: Add information about token order.
• Encoder stack: Builds contextual representations using multi-head self-attention and feed-forward networks.
• Decoder stack: Generates output tokens using masked self-attention, encoder-decoder attention, and feed-forward networks.
• Output layer: Converts decoder representations into vocabulary probabilities.

Each encoder or decoder sublayer uses residual connections and layer normalization. Transformers are effective because they capture long-range dependencies, process sequence elements in parallel during training, and scale efficiently to large datasets and model sizes.

Self-attention allows every token in a sequence to compute a weighted combination of all tokens in the same sequence.

For an input matrix , three projections are calculated:

The compatibility between each query and key is computed using a dot product. The scores are scaled by the square root of the key dimension and normalized with softmax:

Here:

• contains queries.
• contains keys.
• contains values.
• is the dimension of each key vector.

Scaling by prevents large dot products from pushing softmax into regions with very small gradients. The resulting attention weights determine how strongly each token uses information from every other token.

Single-head attention computes one attention distribution using a single set of query, key, and value projections. It may focus mainly on one type of relationship between tokens.

Multi-head attention performs attention in parallel through multiple independently learned heads:

The head outputs are concatenated and projected:

Its advantages include:

• Different heads can learn syntactic, semantic, and positional relationships.
• The model can attend to information from multiple representation subspaces.
• Long-range and local dependencies can be captured simultaneously.

Thus, multi-head attention provides richer representations than a single attention operation, although it requires additional projection parameters and computation.

Self-attention does not inherently preserve sequence order because it processes all token positions in parallel. Positional encoding injects information about the location and relative order of tokens.

In sinusoidal positional encoding, position and dimension are represented as:

The positional vector is added to the token embedding before it enters the Transformer.

Important properties are:

• Different frequencies represent order at different scales.
• Relative positions can be learned from combinations of sinusoidal values.
• No additional parameters are required.
• The encoding can potentially generalize to sequence lengths not encountered during training.

Some modern models instead use learned positional embeddings or relative positional representations.

A Transformer encoder block receives a sequence of contextual vectors and produces an updated sequence of the same length.

Its main operations are:

1. Multi-head self-attention: Every token attends to all permitted tokens in the input sequence.
2. Residual connection and layer normalization: The attention output is combined with the sublayer input and normalized.
3. Position-wise feed-forward network: The same two-layer network is independently applied to every position:

4. Second residual connection and layer normalization: The feed-forward output is combined with its input and normalized.

Dropout is commonly applied for regularization. Stacking encoder blocks progressively creates deep bidirectional representations that capture both local and long-range context.

A Transformer decoder block generates contextual representations for an output sequence. It normally contains three major sublayers:

• Masked multi-head self-attention: A causal mask prevents a position from attending to future output tokens. This preserves autoregressive generation.
• Encoder-decoder cross-attention: Decoder states act as queries, while encoder outputs supply keys and values. This allows generation to use information from the source sequence.
• Position-wise feed-forward network: Applies nonlinear transformations independently to each position.

Each sublayer is accompanied by a residual connection, layer normalization, and usually dropout.

During training, the decoder can process shifted target tokens in parallel because the causal mask blocks unavailable future information. During inference, it generates one token at a time, repeatedly using previously generated tokens and the encoder representation.

Tokenization converts raw text into units that can be mapped to numeric token identifiers and processed by a model.

• Word-level tokenization: Treats each word as one token. It produces meaningful units but requires a very large vocabulary and handles unseen words poorly.
• Character-level tokenization: Uses individual characters. Its vocabulary is small and unknown words are avoided, but sequences become long and individual tokens carry limited semantic information.
• Subword tokenization: Divides words into frequent words and reusable fragments. It balances vocabulary size, sequence length, and coverage.

For example, an uncommon word such as unhappiness might be represented as un, happi, and ness. Pretrained Transformers commonly use subword methods because they can represent rare words, misspellings, and morphological variants without storing every complete word in the vocabulary.

Byte-Pair Encoding (BPE) is a data-driven tokenization method that creates subwords by repeatedly merging frequent adjacent symbols.

The training procedure is:

1. Split vocabulary words into characters or byte-level symbols.
2. Count all adjacent symbol pairs in the training corpus.
3. Merge the most frequent pair into a new symbol.
4. Recalculate pair frequencies.
5. Repeat until the required vocabulary size or number of merges is reached.

For example, if l o w and l o w e r occur frequently, the pair l o may first become lo, followed by a merge of lo w into low.

BPE preserves frequent words as whole tokens while dividing rare words into smaller units. It reduces out-of-vocabulary problems and keeps the vocabulary manageable, although token boundaries do not always correspond to linguistically meaningful morphemes.

BPE and WordPiece both construct subword vocabularies through iterative merging, but their merge criteria differ.

• BPE: Usually merges the most frequent adjacent symbol pair.
• WordPiece: Selects merges that improve a language-model-style likelihood or score, favoring pairs whose joint occurrence is informative relative to their individual frequencies.
• Token notation: WordPiece implementations often mark continuation pieces, such as play and ##ing.
• Usage: BPE variants are used by models such as GPT, while the original BERT commonly uses WordPiece.

Both methods:

• Limit vocabulary size.
• Represent rare or unseen words using known fragments.
• Reduce dependence on an unknown-token symbol.
• Capture recurring prefixes, stems, and suffixes when supported by data.

Their exact output depends on corpus statistics, normalization rules, vocabulary size, and implementation details.

A pretrained transformer model is first trained on a large general-purpose text corpus using a self-supervised objective and then adapted to one or more downstream tasks.

Pretraining stage:

• Uses large quantities of unlabeled text.
• Learns grammar, contextual relationships, semantics, and some factual patterns.
• Applies objectives such as masked language modeling, causal language modeling, or sequence-to-sequence denoising.
• Produces reusable model parameters and representations.

Downstream adaptation stage:

• A task-specific output layer or task formulation is introduced.
• The model is fine-tuned using a smaller labeled dataset, or prompted without full parameter updates.
• Parameters may be fully updated, partially frozen, or modified through parameter-efficient methods.

This approach is effective because general linguistic knowledge is learned once and transferred, reducing the labeled data and training time required for individual NLP tasks.

BERT, or Bidirectional Encoder Representations from Transformers, is an encoder-only pretrained Transformer. Its self-attention layers use both left and right context to build representations.

BERT input representations combine:

• Token embeddings for WordPiece tokens.
• Positional embeddings for sequence positions.
• Segment embeddings to distinguish sentence A from sentence B.
• Special tokens such as [CLS] and [SEP].

Its original pretraining objectives are:

• Masked language modeling (MLM): Predict selected hidden tokens using bidirectional context.
• Next sentence prediction (NSP): Predict whether one segment naturally follows another.

For classification, the final [CLS] representation is commonly passed to a classifier. For token-level tasks, each token representation can be classified separately. Because BERT is an encoder rather than an autoregressive decoder, it is particularly suitable for language understanding tasks.

GPT, or Generative Pretrained Transformer, is primarily a decoder-only Transformer trained to predict the next token from previous tokens.

For a token sequence , causal language modeling factorizes its probability as:

A causal attention mask ensures that position can attend only to positions up to , preventing access to future tokens during training.

During generation:

1. A prompt is tokenized and processed.
2. The model predicts a probability distribution for the next token.
3. A token is selected using greedy decoding, beam search, or sampling.
4. The selected token is appended to the context.
5. The process repeats until a stopping condition is reached.

This objective makes GPT naturally suited to completion, dialogue, summarization, and other tasks formulated as text generation.

T5, or Text-to-Text Transfer Transformer, formulates every NLP task as mapping an input text sequence to an output text sequence. Task prefixes can identify the required operation, such as translate, summarize, or question.

Architectural comparison:

• BERT: Encoder-only and bidirectional; mainly produces representations for understanding tasks.
• GPT: Decoder-only and causal; predicts subsequent tokens for autoregressive generation.
• T5: Encoder-decoder; the encoder reads the complete input and the decoder generates the target sequence.

T5 is pretrained with a denoising objective called span corruption. Consecutive spans of input tokens are replaced by sentinel tokens, and the decoder reconstructs the missing spans.

The unified text-to-text interface allows classification, translation, summarization, and question answering to use the same model, training objective, and output mechanism. For classification, even a class label is generated as text.

In masked language modeling (MLM), selected input tokens are hidden or corrupted, and the model predicts their original identities using surrounding context.

If token is masked, the objective maximizes:

Advantages include:

• It learns bidirectional context.
• It uses unlabeled text for self-supervised pretraining.
• It produces strong representations for classification, named entity recognition, and question answering.
• It can model relationships between distant words.

Limitations include:

• Only selected tokens directly contribute to the prediction loss.
• Artificial mask symbols may create a mismatch because they do not usually appear during fine-tuning.
• MLM is not naturally aligned with left-to-right text generation.
• Predictions for multiple masked positions may be conditionally dependent but are often made in parallel.

BERT is the best-known model trained with this objective.

Next sentence prediction (NSP) is a binary pretraining objective used in the original BERT model.

A training example contains two segments:

• IsNext: Segment B is the actual sentence following segment A in the corpus.
• NotNext: Segment B is sampled from another location and does not naturally follow segment A.

The final representation of the [CLS] token is used to predict the label. NSP was intended to teach relationships between sentence pairs for tasks such as natural language inference and question answering.

However, later research found limitations:

• Random negative sentences may make the task too easy because topic differences reveal the answer.
• The objective mixes topic prediction with discourse coherence.
• Some models, such as RoBERTa, achieved strong results without NSP.

Other approaches use sentence-order prediction, contrastive objectives, or improved document-level pretraining instead.

Masked language modeling (MLM) and causal language modeling (CLM) differ in context access and intended use.

Masked language modeling:

• Predicts deliberately hidden tokens.
• Uses context on both sides of a masked position.
• Is associated with encoder models such as BERT.
• Is well suited to language understanding and representation learning.
• Has a possible pretraining-to-fine-tuning mismatch due to mask tokens.

Causal language modeling:

• Predicts each next token from preceding tokens.
• Uses a causal mask that blocks future positions.
• Is associated with decoder models such as GPT.
• Is naturally suited to autoregressive generation.
• Provides a training signal at nearly every sequence position.

MLM emphasizes deep bidirectional representations, whereas CLM directly learns the probability distribution needed for left-to-right text generation.

Transfer learning reuses knowledge learned from a large source corpus or task to improve performance on a target NLP task.

Common adaptation strategies include:

• Feature extraction: Freeze the pretrained model and use its representations as input to a separate classifier.
• Full fine-tuning: Update all pretrained parameters and the task-specific head.
• Partial fine-tuning: Freeze lower layers and update only selected upper layers.
• Parameter-efficient fine-tuning: Train small adapters, prompts, or low-rank updates while keeping most original parameters fixed.
• Prompt-based adaptation: Express the task through instructions or demonstrations, with little or no parameter updating.

Benefits include reduced labeled-data requirements, faster convergence, and improved generalization. Risks include catastrophic forgetting, overfitting on small datasets, domain mismatch, and high memory costs. Suitable learning rates, validation, regularization, and domain-relevant data help control these risks.

To fine-tune a pretrained Transformer for text classification, the following procedure is commonly used:

1. Tokenize each document using the model's tokenizer.
2. Add required special tokens and create attention masks.
3. Pad or truncate examples to a suitable sequence length.
4. Pass the batch through the pretrained model.
5. Extract a pooled sequence representation, such as BERT's [CLS] output.
6. Feed it to a classification layer that produces one logit per class.
7. Minimize cross-entropy loss for single-label classification:

8. Update the classification head and usually the Transformer parameters using a small learning rate.

Evaluation may use accuracy, precision, recall, and score. For multi-label classification, independent sigmoid outputs and binary cross-entropy are typically used instead of softmax.

Named entity recognition (NER) is formulated as token classification. Each input token receives a label such as B-PER, I-PER, B-ORG, or O.

The fine-tuning process is:

• Tokenize the sentence using the pretrained model's subword tokenizer.
• Obtain a contextual representation for every subword.
• Apply a shared linear classification layer to each representation.
• Compute token-level cross-entropy loss over labeled positions.
• Decode the predicted labels, optionally enforcing valid BIO transitions.

A key issue is subword-label alignment. One original word may be divided into several pieces. Common solutions are:

• Assign the word's label only to the first subword and ignore the remaining pieces in the loss.
• Copy or adapt the label across all pieces, such as assigning B-ORG to the first piece and I-ORG to later pieces.

Predictions must finally be mapped back from subwords to the original words or character spans.

In extractive question answering, the answer is predicted as a contiguous span within a supplied context.

The question and context are tokenized together. A pretrained encoder produces a representation for every token. Two output projections calculate start and end logits:

Softmax distributions over these logits estimate the answer's start and end positions. Training minimizes the sum of start-position and end-position cross-entropy losses. During inference, the system selects a valid high-scoring pair satisfying .

Using Hugging Face Transformers, a typical workflow is:

• Load AutoTokenizer and AutoModelForQuestionAnswering from a checkpoint.
• Tokenize question-context pairs with truncation and attention masks.
• Use sliding windows and stride for contexts longer than the model limit.
• Train with Trainer or a custom PyTorch loop.
• Map token spans back to context characters using offset mappings.
• Evaluate with exact match and token-level .

Hugging Face also provides pipelines for concise inference, but careful preprocessing is still necessary for long contexts and batched evaluation.