Back to Subject

Unit 5: Transformers and Pretrained Language Models - Practice Quiz

CSE472 — Deep Learning For Natural Language Processing 60 Questions
0 Correct 0 Wrong 60 Left
0/60

1 What mechanism allows a Transformer to model relationships between tokens without using recurrence?

Transformer architecture Easy
A. Max pooling
B. Gradient clipping
C. Self-attention
D. Convolution

2 In self-attention, what does each token attend to?

Self-attention Easy
A. Only the final token
B. Only the previous token
C. Tokens in the same sequence
D. Tokens in another dataset

3 What is the main purpose of using multiple attention heads?

Multi-head attention Easy
A. To replace positional encoding
B. To remove token embeddings
C. To shorten every sentence
D. To learn different relationships

4 Why is positional encoding added to token embeddings in a Transformer?

Positional encoding Easy
A. To reduce vocabulary size
B. To indicate token order
C. To remove rare words
D. To label training examples

5 Which Transformer component typically processes the input sequence into contextual representations?

Transformer encoder and decoder blocks Easy
A. Encoder stack
B. Loss function
C. Vocabulary index
D. Output tokenizer

6 What is the purpose of tokenization in NLP?

Tokenization methods Easy
A. To remove every punctuation mark
B. To translate text automatically
C. To divide text into units
D. To calculate model accuracy

7 How does Byte-Pair Encoding commonly build subword tokens?

Byte-Pair Encoding Easy
A. By translating words into labels
B. By sorting words alphabetically
C. By merging frequent symbol pairs
D. By deleting frequent word pairs

8 What kind of units does WordPiece primarily use to represent text?

WordPiece Easy
A. Audio frames
B. Sentence labels
C. Image patches
D. Subword pieces

9 What does it mean when a Transformer model is pretrained?

Pretrained transformer models Easy
A. It runs without tokenization
B. It has learned from a large corpus
C. It supports only one sentence
D. It has no learned parameters

10 Which Transformer structure is primarily used by the original BERT model?

BERT Easy
A. Recurrent-only structure
B. Encoder-only structure
C. Convolution-only structure
D. Decoder-only structure

11 Which task is GPT especially well suited for?

GPT Easy
A. Autoregressive text generation
B. Audio noise removal
C. Database table sorting
D. Image edge detection

12 How does T5 represent many NLP tasks?

T5 Easy
A. As audio-only problems
B. As graph-only problems
C. As text-to-text problems
D. As image-to-image problems

13 What does a model learn to do during masked language modeling?

Masked language modeling Easy
A. Count all input sentences
B. Generate class identifiers
C. Remove duplicate documents
D. Predict hidden input tokens

14 What does next sentence prediction ask a model to determine?

Next sentence prediction Easy
A. Whether two words share a prefix
B. Whether a document has a title
C. Whether one sentence follows another
D. Whether a sentence contains punctuation

15 In causal language modeling, which information is used to predict the next token?

Causal language modeling Easy
A. External images
B. Only future tokens
C. Previous tokens
D. Random class labels

16 What is transfer learning in NLP?

Transfer learning for NLP tasks Easy
A. Training without any existing parameters
B. Converting every word into a number
C. Moving text between storage devices
D. Applying learned knowledge to a new task

17 What is typically added to a pretrained Transformer for text classification?

Fine-tuning for text classification Easy
A. A classification head
B. An image filter
C. A sorting algorithm
D. A speech decoder

18 What does named entity recognition identify in text?

Named entity recognition Easy
A. Pixels, edges, and shapes
B. Rows, columns, and formulas
C. People, places, and organizations
D. Fonts, colors, and margins

19 In extractive question answering, where does the answer usually come from?

Question answering Easy
A. A random model parameter
B. A span in the given passage
C. A hidden image region
D. A separate audio recording

20 What does the HuggingFace Transformers library primarily provide?

HuggingFace Transformers Easy
A. Pretrained models and NLP tools
B. Spreadsheet templates and formulas
C. Three-dimensional rendering engines
D. Operating system device drivers

21 A Transformer processes a sequence without using recurrence. Which mechanism allows each token representation to incorporate information from other tokens in the same layer?

Transformer architecture Medium
A. Self-attention over token representations
B. A lookup table of sentence labels
C. A recurrent hidden-state update
D. A fixed convolution over characters

22 In scaled dot-product attention, why is the query-key dot product divided by ?

Self-attention Medium
A. To guarantee equal attention for all tokens
B. To remove positional information from keys
C. To increase the number of attention heads
D. To reduce the variance of large dot products

23 Why can multi-head attention represent relationships that a single attention head might miss?

Multi-head attention Medium
A. Each head receives a completely different input sentence
B. Each head replaces the feed-forward sublayer
C. Each head removes the need for tokenization
D. Each head can learn a different projection and relationship pattern

24 A Transformer receives the same set of token embeddings in two different orders. What is the main purpose of positional encoding in this situation?

Positional encoding Medium
A. To make the representations sensitive to token order
B. To convert words into subword vocabulary entries
C. To prevent attention from using query vectors
D. To assign a unique class label to each position

25 During text generation, which attention pattern is required in a Transformer decoder to prevent a token from using future target tokens?

Transformer encoder and decoder blocks Medium
A. A causal mask on decoder self-attention
B. A padding mask on encoder outputs
C. A larger embedding dimension for target tokens
D. A shared vocabulary for source and target text

26 A tokenizer represents an unfamiliar word using several meaningful or partially meaningful pieces instead of mapping it to one unknown token. What advantage does this provide?

Tokenization methods Medium
A. It guarantees one token per word
B. It reduces out-of-vocabulary problems
C. It removes the need for model training
D. It preserves every word as an unchanged string

27 In Byte-Pair Encoding for subword tokenization, a frequent adjacent symbol pair is repeatedly merged. If t and h frequently occur together, what may happen after training?

Byte-Pair Encoding Medium
A. Every word containing t is deleted
B. The vocabulary is limited to single characters
C. The pair t h may become the token th
D. The pair t h is always replaced by an unknown token

28 A WordPiece tokenizer splits the word "unhappiness" into un, ##happi, and ##ness. What does the ## marker usually indicate?

WordPiece Medium
A. The piece must be converted into an uppercase token
B. The piece represents a sentence boundary
C. The piece continues a word rather than starting one
D. The piece was generated by a character-level model

29 A team has limited labeled data for sentiment analysis but access to a large pretrained Transformer. Which strategy is generally most appropriate?

Pretrained transformer models Medium
A. Freeze all parameters and omit a task-specific prediction layer
B. Use random labels to adapt the output layer
C. Discard the pretrained weights and train only a tokenizer
D. Fine-tune the pretrained model on labeled sentiment examples

30 Why is BERT particularly suitable for classifying a sentence based on both earlier and later context?

BERT Medium
A. Its training objective removes all sentence context
B. Its tokenizer contains only complete words
C. Its encoder uses bidirectional self-attention
D. Its decoder predicts only the next word

31 Which task aligns most directly with the original design of a GPT-style Transformer?

GPT Medium
A. Assigning a label to every input character
B. Predicting masked tokens using both sequence directions
C. Aligning two sentences with no language modeling
D. Generating the next token from preceding tokens

32 A T5 model is fine-tuned to answer questions. How is the task typically represented for the model?

T5 Medium
A. As a sentence pair mapped only to a binary mask
B. As token embeddings mapped directly to attention weights
C. As text input mapped to text output
D. As an image input mapped to a class index

33 During masked language model pretraining, a model sees the sentence "The cat sat on the [MASK]." Which prediction best describes the training target?

Masked language modeling Medium
A. The position of the sentence in the dataset
B. The original token at the masked position
C. The complete document topic label
D. The token that follows the final period

34 In the next sentence prediction objective, what does the model typically learn to determine?

Next sentence prediction Medium
A. Whether a paragraph contains a masked word
B. Whether a token should be split into characters
C. Whether one sentence logically follows another
D. Whether every word is correctly spelled

35 For the token sequence , what conditional factorization is used by a causal language model?

Causal language modeling Medium
A.
B.
C.
D. without conditional terms

36 What is the main benefit of transferring a pretrained language model to a new NLP task?

Transfer learning for NLP tasks Medium
A. It eliminates the need to select an evaluation metric
B. It ensures that all downstream tasks use identical labels
C. It guarantees perfect performance without task data
D. It reuses general linguistic representations learned from broad data

37 A binary text classifier is built with BERT. Which change is normally made to produce the final prediction?

Fine-tuning for text classification Medium
A. Train only the positional encoding values
B. Add a classification head over a sequence representation
C. Replace all attention layers with a single recurrent cell
D. Remove the tokenizer and classify raw characters

38 For named entity recognition, why are predictions usually made for individual tokens rather than only for the entire sentence?

Named entity recognition Medium
A. Token labels prevent the use of pretrained models
B. Sentence-level labels cannot be represented numerically
C. Each token may have a different entity label
D. Every sentence contains exactly one entity

39 In extractive question answering, the answer is contained in a context passage. What does the model typically predict?

Question answering Medium
A. A new vocabulary for the question
B. The start and end positions of the answer span
C. A single sentiment label for the passage
D. The next sentence after the passage

40 When using HuggingFace Transformers for a pretrained model, why should the tokenizer match the model checkpoint?

HuggingFace Transformers Medium
A. It ensures that every input contains the same number of words
B. It preserves the checkpoint's vocabulary and token-ID mapping
C. It guarantees that the model will not require fine-tuning
D. It changes the Transformer into a recurrent network

41 A Transformer processes a sequence of length with hidden size . Ignoring constants and feed-forward layers, which change most directly reduces the asymptotic self-attention cost while preserving ?

Transformer architecture Hard
A. Reducing each head dimension while keeping fixed
B. Reducing the number of heads from to
C. Reducing the sequence length from to
D. Sharing projection matrices across adjacent layers

42 For and , what is the first output row of scaled dot-product self-attention with key dimension and no mask?

Self-attention Hard
A. , where
B. , where
C. , where
D. , where

43 A model has and 12 attention heads. It is changed to 24 heads while all query, key, value, and output projections remain dense matrices. Which statement is correct?

Multi-head attention Hard
A. Each head dimension becomes 32, while projection parameter counts remain unchanged
B. Each head dimension remains 64, while projection parameter counts double
C. Each head dimension remains 64, while projection parameter counts remain unchanged
D. Each head dimension becomes 32, while projection parameter counts are halved

44 Why can sinusoidal positional encodings support reasoning about relative offsets without learning a separate vector for every offset?

Positional encoding Hard
A. Each frequency is learned so that unseen positions interpolate between training positions
B. The encoding makes attention scores invariant to all permutations of the input tokens
C. A fixed linear transformation relates the encoding at to that at for fixed
D. Every relative offset is represented by a unique one-hot subvector shared across positions

45 During teacher-forced training of a standard encoder-decoder Transformer, which masking configuration is required?

Transformer encoder and decoder blocks Hard
A. Encoder padding mask, bidirectional decoder self-attention, and causal cross-attention mask
B. Causal encoder mask, padding decoder self-attention mask, and unmasked cross-attention
C. Encoder padding mask, causal decoder self-attention mask, and source padding cross-attention mask
D. Unmasked encoder attention, causal decoder self-attention, and target padding cross-attention mask

46 A production NLP system must accept arbitrary Unicode input without emitting an unknown-token symbol. Which tokenizer design most directly guarantees this property, assuming valid byte input?

Tokenization methods Hard
A. A word-level tokenizer built from a very large multilingual corpus
B. A morpheme tokenizer with language-specific fallback dictionaries
C. A byte-level tokenizer whose base vocabulary contains every byte value
D. A character tokenizer containing only observed Unicode code points

47 A BPE trainer starts with the segmented corpus low low lower represented as characters plus end-of-word markers. If (l,o) and (o,w) have equal maximum frequency, what is required for independently trained tokenizers to produce identical vocabularies?

Byte-Pair Encoding Hard
A. A deterministic tie-breaking rule and the same ordered training corpus
B. A shared unknown token and the same maximum token length
C. A common normalization form and randomly selected merge ties
D. A larger merge budget and independently shuffled training examples

48 Compared with frequency-only BPE merging, a WordPiece-style merge score favors a pair with high joint frequency relative to its component frequencies. Which pair is therefore most likely to be preferred?

WordPiece Hard
A. , , and
B. , , and
C. , , and
D. , , and

49 A task requires assigning a label using both left and right context around every token, while inference is non-generative. Which pretrained architecture-objective pairing offers the closest inductive match?

Pretrained transformer models Hard
A. A decoder-only Transformer trained with causal token prediction
B. An encoder-only Transformer trained with masked token prediction
C. A decoder-only Transformer trained with left-to-right sentence ordering
D. An encoder-decoder Transformer trained only with prefix continuation

50 In original BERT input construction for sentence-pair classification, what distinguishes tokens from sentence A and sentence B before the encoder layers?

BERT Hard
A. Their positional indices restart at zero when sentence B begins
B. Their token embeddings are projected through separate encoder parameter sets
C. Their attention heads are partitioned into sentence-specific groups
D. Their token embeddings are summed with different learned segment embeddings

51 A decoder-only GPT model is trained on packed documents. Which attention mask prevents both future-token leakage and information flow across document boundaries?

GPT Hard
A. A block-diagonal lower-triangular mask aligned with document boundaries
B. A global bidirectional mask with padding positions removed
C. A global lower-triangular mask spanning every packed document
D. A block-diagonal bidirectional mask aligned with document boundaries

52 T5 pretraining replaces multiple contiguous spans with sentinel tokens. Which target format is consistent with its span-corruption objective?

T5 Hard
A. Sentinel token, first missing span, next sentinel token, second missing span
B. One sentinel token followed by every uncorrupted token from the input
C. All missing tokens in original order without any sentinel boundary tokens
D. Original input sequence with corrupted spans replaced by ordinary mask tokens

53 In BERT-style masking, selected tokens are not always replaced by [MASK]; some remain unchanged or become random tokens. What mismatch is this strategy primarily intended to reduce?

Masked language modeling Hard
A. The difference between bidirectional and causal attention computation
B. The imbalance between positive and negative next-sentence examples
C. The variation in subword vocabulary size across downstream datasets
D. The absence of the artificial [MASK] token during downstream inference

54 Why can randomly sampled negative pairs make next sentence prediction solvable through shortcuts rather than discourse understanding?

Next sentence prediction Hard
A. Negative sentences may differ in topic or document style from positive continuations
B. Positive continuations are encoded causally while negative pairs are bidirectional
C. Positive pairs always contain fewer subword tokens than randomly sampled pairs
D. Negative pairs necessarily reverse the grammatical order of both sentences

55 A causal language model assigns token probabilities , , and to the three observed next tokens. Using natural logarithms, what is the mean token negative log-likelihood?

Causal language modeling Hard
A.
B.
C.
D.

56 A pretrained encoder performs well on general text but overfits a 500-example legal dataset during full fine-tuning. Which intervention most directly reduces trainable capacity while retaining task adaptation?

Transfer learning for NLP tasks Hard
A. Freeze the encoder and train a task-specific classification head
B. Unfreeze every layer and increase the learning rate uniformly
C. Expand the hidden size and preserve the original dropout rate
D. Reinitialize the encoder and train all parameters from scratch

57 A classifier truncates every document to the first 512 tokens, but decisive evidence often appears near the end. Which change addresses the information-loss mechanism most directly?

Fine-tuning for text classification Hard
A. Apply label smoothing while preserving the same truncated input
B. Increase dropout in the classification head while keeping first-token truncation
C. Classify overlapping chunks and aggregate their document-level evidence
D. Lower the encoder learning rate while keeping first-token truncation

58 The word Washington is split into Wash, ##ing, and ##ton. Under a first-subtoken labeling scheme, how should loss and prediction alignment usually be handled?

Named entity recognition Hard
A. Assign the entity label to Wash and ignore the other subtokens in the loss
B. Assign B-LOC independently to all three subtokens and score every position
C. Assign the entity label only to ##ton and ignore the preceding subtokens
D. Assign B-LOC, I-LOC, I-LOC and treat them as three gold words

59 In extractive QA, a model independently scores start and end positions. Which decoding rule avoids invalid spans while respecting a maximum answer length ?

Question answering Hard
A. Choose the maximum start and maximum end independently, then reorder them
B. Choose the shortest span containing either the best start or best end position
C. Maximize subject to and
D. Maximize subject only to $i
e j$ and no length restriction

60 A HuggingFace token-classification pipeline pads batches to different lengths. Which setup ensures padded positions neither affect encoder attention nor contribute to cross-entropy loss?

HuggingFace Transformers Hard
A. Set attention_mask to 1 and labels to 0 at padded positions
B. Set attention_mask to 0 and labels to -100 at padded positions
C. Set input_ids to 0 and labels to the outside tag at padded positions
D. Set token_type_ids to 0 and labels to -1 at padded positions
Previous MCQ
Unit 4
Next MCQ
Unit 6