Back to Subject

Unit 5: Transformers and Pretrained Language Models - Practice Quiz

CSE472 — Deep Learning For Natural Language Processing 60 Questions
0 Correct 0 Wrong 60 Left
0/60

1 What mechanism allows a Transformer to model relationships between tokens without using recurrence?

Transformer architecture Easy
A. Self-attention
B. Gradient clipping
C. Max pooling
D. Convolution

2 In self-attention, what does each token attend to?

Self-attention Easy
A. Only the final token
B. Tokens in the same sequence
C. Tokens in another dataset
D. Only the previous token

3 What is the main purpose of using multiple attention heads?

Multi-head attention Easy
A. To learn different relationships
B. To remove token embeddings
C. To replace positional encoding
D. To shorten every sentence

4 Why is positional encoding added to token embeddings in a Transformer?

Positional encoding Easy
A. To reduce vocabulary size
B. To remove rare words
C. To label training examples
D. To indicate token order

5 Which Transformer component typically processes the input sequence into contextual representations?

Transformer encoder and decoder blocks Easy
A. Output tokenizer
B. Loss function
C. Vocabulary index
D. Encoder stack

6 What is the purpose of tokenization in NLP?

Tokenization methods Easy
A. To divide text into units
B. To translate text automatically
C. To remove every punctuation mark
D. To calculate model accuracy

7 How does Byte-Pair Encoding commonly build subword tokens?

Byte-Pair Encoding Easy
A. By deleting frequent word pairs
B. By translating words into labels
C. By sorting words alphabetically
D. By merging frequent symbol pairs

8 What kind of units does WordPiece primarily use to represent text?

WordPiece Easy
A. Subword pieces
B. Image patches
C. Sentence labels
D. Audio frames

9 What does it mean when a Transformer model is pretrained?

Pretrained transformer models Easy
A. It runs without tokenization
B. It supports only one sentence
C. It has learned from a large corpus
D. It has no learned parameters

10 Which Transformer structure is primarily used by the original BERT model?

BERT Easy
A. Encoder-only structure
B. Convolution-only structure
C. Recurrent-only structure
D. Decoder-only structure

11 Which task is GPT especially well suited for?

GPT Easy
A. Database table sorting
B. Image edge detection
C. Autoregressive text generation
D. Audio noise removal

12 How does T5 represent many NLP tasks?

T5 Easy
A. As graph-only problems
B. As text-to-text problems
C. As audio-only problems
D. As image-to-image problems

13 What does a model learn to do during masked language modeling?

Masked language modeling Easy
A. Count all input sentences
B. Generate class identifiers
C. Predict hidden input tokens
D. Remove duplicate documents

14 What does next sentence prediction ask a model to determine?

Next sentence prediction Easy
A. Whether a document has a title
B. Whether one sentence follows another
C. Whether a sentence contains punctuation
D. Whether two words share a prefix

15 In causal language modeling, which information is used to predict the next token?

Causal language modeling Easy
A. Only future tokens
B. Random class labels
C. Previous tokens
D. External images

16 What is transfer learning in NLP?

Transfer learning for NLP tasks Easy
A. Applying learned knowledge to a new task
B. Training without any existing parameters
C. Moving text between storage devices
D. Converting every word into a number

17 What is typically added to a pretrained Transformer for text classification?

Fine-tuning for text classification Easy
A. An image filter
B. A classification head
C. A speech decoder
D. A sorting algorithm

18 What does named entity recognition identify in text?

Named entity recognition Easy
A. People, places, and organizations
B. Rows, columns, and formulas
C. Fonts, colors, and margins
D. Pixels, edges, and shapes

19 In extractive question answering, where does the answer usually come from?

Question answering Easy
A. A span in the given passage
B. A random model parameter
C. A hidden image region
D. A separate audio recording

20 What does the HuggingFace Transformers library primarily provide?

HuggingFace Transformers Easy
A. Pretrained models and NLP tools
B. Operating system device drivers
C. Spreadsheet templates and formulas
D. Three-dimensional rendering engines

21 A Transformer processes a sequence without using recurrence. Which mechanism allows each token representation to incorporate information from other tokens in the same layer?

Transformer architecture Medium
A. A fixed convolution over characters
B. A lookup table of sentence labels
C. A recurrent hidden-state update
D. Self-attention over token representations

22 In scaled dot-product attention, why is the query-key dot product divided by ?

Self-attention Medium
A. To reduce the variance of large dot products
B. To guarantee equal attention for all tokens
C. To increase the number of attention heads
D. To remove positional information from keys

23 Why can multi-head attention represent relationships that a single attention head might miss?

Multi-head attention Medium
A. Each head removes the need for tokenization
B. Each head can learn a different projection and relationship pattern
C. Each head replaces the feed-forward sublayer
D. Each head receives a completely different input sentence

24 A Transformer receives the same set of token embeddings in two different orders. What is the main purpose of positional encoding in this situation?

Positional encoding Medium
A. To assign a unique class label to each position
B. To make the representations sensitive to token order
C. To prevent attention from using query vectors
D. To convert words into subword vocabulary entries

25 During text generation, which attention pattern is required in a Transformer decoder to prevent a token from using future target tokens?

Transformer encoder and decoder blocks Medium
A. A shared vocabulary for source and target text
B. A causal mask on decoder self-attention
C. A padding mask on encoder outputs
D. A larger embedding dimension for target tokens

26 A tokenizer represents an unfamiliar word using several meaningful or partially meaningful pieces instead of mapping it to one unknown token. What advantage does this provide?

Tokenization methods Medium
A. It removes the need for model training
B. It preserves every word as an unchanged string
C. It reduces out-of-vocabulary problems
D. It guarantees one token per word

27 In Byte-Pair Encoding for subword tokenization, a frequent adjacent symbol pair is repeatedly merged. If t and h frequently occur together, what may happen after training?

Byte-Pair Encoding Medium
A. The pair t h is always replaced by an unknown token
B. The vocabulary is limited to single characters
C. The pair t h may become the token th
D. Every word containing t is deleted

28 A WordPiece tokenizer splits the word "unhappiness" into un, ##happi, and ##ness. What does the ## marker usually indicate?

WordPiece Medium
A. The piece represents a sentence boundary
B. The piece continues a word rather than starting one
C. The piece was generated by a character-level model
D. The piece must be converted into an uppercase token

29 A team has limited labeled data for sentiment analysis but access to a large pretrained Transformer. Which strategy is generally most appropriate?

Pretrained transformer models Medium
A. Use random labels to adapt the output layer
B. Fine-tune the pretrained model on labeled sentiment examples
C. Freeze all parameters and omit a task-specific prediction layer
D. Discard the pretrained weights and train only a tokenizer

30 Why is BERT particularly suitable for classifying a sentence based on both earlier and later context?

BERT Medium
A. Its training objective removes all sentence context
B. Its encoder uses bidirectional self-attention
C. Its tokenizer contains only complete words
D. Its decoder predicts only the next word

31 Which task aligns most directly with the original design of a GPT-style Transformer?

GPT Medium
A. Assigning a label to every input character
B. Generating the next token from preceding tokens
C. Aligning two sentences with no language modeling
D. Predicting masked tokens using both sequence directions

32 A T5 model is fine-tuned to answer questions. How is the task typically represented for the model?

T5 Medium
A. As text input mapped to text output
B. As a sentence pair mapped only to a binary mask
C. As an image input mapped to a class index
D. As token embeddings mapped directly to attention weights

33 During masked language model pretraining, a model sees the sentence "The cat sat on the [MASK]." Which prediction best describes the training target?

Masked language modeling Medium
A. The complete document topic label
B. The position of the sentence in the dataset
C. The token that follows the final period
D. The original token at the masked position

34 In the next sentence prediction objective, what does the model typically learn to determine?

Next sentence prediction Medium
A. Whether one sentence logically follows another
B. Whether a paragraph contains a masked word
C. Whether every word is correctly spelled
D. Whether a token should be split into characters

35 For the token sequence , what conditional factorization is used by a causal language model?

Causal language modeling Medium
A.
B. without conditional terms
C.
D.

36 What is the main benefit of transferring a pretrained language model to a new NLP task?

Transfer learning for NLP tasks Medium
A. It guarantees perfect performance without task data
B. It reuses general linguistic representations learned from broad data
C. It ensures that all downstream tasks use identical labels
D. It eliminates the need to select an evaluation metric

37 A binary text classifier is built with BERT. Which change is normally made to produce the final prediction?

Fine-tuning for text classification Medium
A. Train only the positional encoding values
B. Replace all attention layers with a single recurrent cell
C. Add a classification head over a sequence representation
D. Remove the tokenizer and classify raw characters

38 For named entity recognition, why are predictions usually made for individual tokens rather than only for the entire sentence?

Named entity recognition Medium
A. Every sentence contains exactly one entity
B. Each token may have a different entity label
C. Token labels prevent the use of pretrained models
D. Sentence-level labels cannot be represented numerically

39 In extractive question answering, the answer is contained in a context passage. What does the model typically predict?

Question answering Medium
A. The next sentence after the passage
B. A new vocabulary for the question
C. The start and end positions of the answer span
D. A single sentiment label for the passage

40 When using HuggingFace Transformers for a pretrained model, why should the tokenizer match the model checkpoint?

HuggingFace Transformers Medium
A. It changes the Transformer into a recurrent network
B. It ensures that every input contains the same number of words
C. It preserves the checkpoint's vocabulary and token-ID mapping
D. It guarantees that the model will not require fine-tuning

41 A Transformer processes a sequence of length with hidden size . Ignoring constants and feed-forward layers, which change most directly reduces the asymptotic self-attention cost while preserving ?

Transformer architecture Hard
A. Reducing the sequence length from to
B. Sharing projection matrices across adjacent layers
C. Reducing each head dimension while keeping fixed
D. Reducing the number of heads from to

42 For and , what is the first output row of scaled dot-product self-attention with key dimension and no mask?

Self-attention Hard
A. , where
B. , where
C. , where
D. , where

43 A model has and 12 attention heads. It is changed to 24 heads while all query, key, value, and output projections remain dense matrices. Which statement is correct?

Multi-head attention Hard
A. Each head dimension becomes 32, while projection parameter counts remain unchanged
B. Each head dimension remains 64, while projection parameter counts double
C. Each head dimension remains 64, while projection parameter counts remain unchanged
D. Each head dimension becomes 32, while projection parameter counts are halved

44 Why can sinusoidal positional encodings support reasoning about relative offsets without learning a separate vector for every offset?

Positional encoding Hard
A. The encoding makes attention scores invariant to all permutations of the input tokens
B. Each frequency is learned so that unseen positions interpolate between training positions
C. A fixed linear transformation relates the encoding at to that at for fixed
D. Every relative offset is represented by a unique one-hot subvector shared across positions

45 During teacher-forced training of a standard encoder-decoder Transformer, which masking configuration is required?

Transformer encoder and decoder blocks Hard
A. Encoder padding mask, bidirectional decoder self-attention, and causal cross-attention mask
B. Encoder padding mask, causal decoder self-attention mask, and source padding cross-attention mask
C. Causal encoder mask, padding decoder self-attention mask, and unmasked cross-attention
D. Unmasked encoder attention, causal decoder self-attention, and target padding cross-attention mask

46 A production NLP system must accept arbitrary Unicode input without emitting an unknown-token symbol. Which tokenizer design most directly guarantees this property, assuming valid byte input?

Tokenization methods Hard
A. A byte-level tokenizer whose base vocabulary contains every byte value
B. A word-level tokenizer built from a very large multilingual corpus
C. A character tokenizer containing only observed Unicode code points
D. A morpheme tokenizer with language-specific fallback dictionaries

47 A BPE trainer starts with the segmented corpus low low lower represented as characters plus end-of-word markers. If (l,o) and (o,w) have equal maximum frequency, what is required for independently trained tokenizers to produce identical vocabularies?

Byte-Pair Encoding Hard
A. A deterministic tie-breaking rule and the same ordered training corpus
B. A larger merge budget and independently shuffled training examples
C. A common normalization form and randomly selected merge ties
D. A shared unknown token and the same maximum token length

48 Compared with frequency-only BPE merging, a WordPiece-style merge score favors a pair with high joint frequency relative to its component frequencies. Which pair is therefore most likely to be preferred?

WordPiece Hard
A. , , and
B. , , and
C. , , and
D. , , and

49 A task requires assigning a label using both left and right context around every token, while inference is non-generative. Which pretrained architecture-objective pairing offers the closest inductive match?

Pretrained transformer models Hard
A. An encoder-only Transformer trained with masked token prediction
B. A decoder-only Transformer trained with left-to-right sentence ordering
C. An encoder-decoder Transformer trained only with prefix continuation
D. A decoder-only Transformer trained with causal token prediction

50 In original BERT input construction for sentence-pair classification, what distinguishes tokens from sentence A and sentence B before the encoder layers?

BERT Hard
A. Their token embeddings are summed with different learned segment embeddings
B. Their token embeddings are projected through separate encoder parameter sets
C. Their attention heads are partitioned into sentence-specific groups
D. Their positional indices restart at zero when sentence B begins

51 A decoder-only GPT model is trained on packed documents. Which attention mask prevents both future-token leakage and information flow across document boundaries?

GPT Hard
A. A global bidirectional mask with padding positions removed
B. A block-diagonal bidirectional mask aligned with document boundaries
C. A global lower-triangular mask spanning every packed document
D. A block-diagonal lower-triangular mask aligned with document boundaries

52 T5 pretraining replaces multiple contiguous spans with sentinel tokens. Which target format is consistent with its span-corruption objective?

T5 Hard
A. Sentinel token, first missing span, next sentinel token, second missing span
B. All missing tokens in original order without any sentinel boundary tokens
C. Original input sequence with corrupted spans replaced by ordinary mask tokens
D. One sentinel token followed by every uncorrupted token from the input

53 In BERT-style masking, selected tokens are not always replaced by [MASK]; some remain unchanged or become random tokens. What mismatch is this strategy primarily intended to reduce?

Masked language modeling Hard
A. The variation in subword vocabulary size across downstream datasets
B. The absence of the artificial [MASK] token during downstream inference
C. The difference between bidirectional and causal attention computation
D. The imbalance between positive and negative next-sentence examples

54 Why can randomly sampled negative pairs make next sentence prediction solvable through shortcuts rather than discourse understanding?

Next sentence prediction Hard
A. Negative pairs necessarily reverse the grammatical order of both sentences
B. Positive pairs always contain fewer subword tokens than randomly sampled pairs
C. Positive continuations are encoded causally while negative pairs are bidirectional
D. Negative sentences may differ in topic or document style from positive continuations

55 A causal language model assigns token probabilities , , and to the three observed next tokens. Using natural logarithms, what is the mean token negative log-likelihood?

Causal language modeling Hard
A.
B.
C.
D.

56 A pretrained encoder performs well on general text but overfits a 500-example legal dataset during full fine-tuning. Which intervention most directly reduces trainable capacity while retaining task adaptation?

Transfer learning for NLP tasks Hard
A. Expand the hidden size and preserve the original dropout rate
B. Reinitialize the encoder and train all parameters from scratch
C. Freeze the encoder and train a task-specific classification head
D. Unfreeze every layer and increase the learning rate uniformly

57 A classifier truncates every document to the first 512 tokens, but decisive evidence often appears near the end. Which change addresses the information-loss mechanism most directly?

Fine-tuning for text classification Hard
A. Lower the encoder learning rate while keeping first-token truncation
B. Increase dropout in the classification head while keeping first-token truncation
C. Classify overlapping chunks and aggregate their document-level evidence
D. Apply label smoothing while preserving the same truncated input

58 The word Washington is split into Wash, ##ing, and ##ton. Under a first-subtoken labeling scheme, how should loss and prediction alignment usually be handled?

Named entity recognition Hard
A. Assign the entity label only to ##ton and ignore the preceding subtokens
B. Assign B-LOC independently to all three subtokens and score every position
C. Assign B-LOC, I-LOC, I-LOC and treat them as three gold words
D. Assign the entity label to Wash and ignore the other subtokens in the loss

59 In extractive QA, a model independently scores start and end positions. Which decoding rule avoids invalid spans while respecting a maximum answer length ?

Question answering Hard
A. Choose the shortest span containing either the best start or best end position
B. Maximize subject to and
C. Maximize subject only to $i
e j$ and no length restriction
D. Choose the maximum start and maximum end independently, then reorder them

60 A HuggingFace token-classification pipeline pads batches to different lengths. Which setup ensures padded positions neither affect encoder attention nor contribute to cross-entropy loss?

HuggingFace Transformers Hard
A. Set token_type_ids to 0 and labels to -1 at padded positions
B. Set input_ids to 0 and labels to the outside tag at padded positions
C. Set attention_mask to 0 and labels to -100 at padded positions
D. Set attention_mask to 1 and labels to 0 at padded positions
Previous MCQ
Unit 4
Next MCQ
Unit 6