Unit 6 - Practice Quiz

1 What is the primary goal of a generative NLP model?

2 In autoregressive text generation, how is a sequence typically generated?

3 How does the greedy search decoding strategy choose the next token?

4 What is a common drawback of using greedy search for text generation?

5 Which of the following best describes beam search?

6 What happens during top-k sampling in text generation?

7 If in top-k sampling, the strategy becomes exactly equivalent to which other decoding method?

8 What is another common name for nucleus sampling?

9 Unlike top-k sampling which uses a fixed number of tokens, how does nucleus (top-p) sampling determine the pool of candidate tokens?

10 What does it mean for a large language model to be 'instruction-tuned'?

11 Which of the following is a primary characteristic of Large Language Models (LLMs)?

12 When an LLM performs abstractive summarization, what is it primarily doing?

13 What is a critical requirement for a model to perform well in multi-turn dialogue generation?

14 Which prompting technique explicitly encourages an LLM to generate intermediate logical steps to solve complex reasoning tasks?

15 Which automated metric is widely used in NLP for evaluating the n-gram overlap between generated text and reference text?

16 In the context of evaluating a language model, what does a lower perplexity score indicate?

17 Why are human judgment measures often considered essential for evaluating generative LLMs?

18 What does the term 'hallucination' refer to in the context of Large Language Models?

19 Why is 'explainability' a major challenge in modern Large Language Models?

20 Which neural network architecture serves as the foundation for modern generative LLMs like GPT?

21 Which of the following best describes the core mathematical objective of standard autoregressive generative NLP models during generation?

22 When generating text, modifying the 'temperature' parameter alters the probability distribution of the next token. If a temperature is applied, how does it affect the softmax output?

23 A generative model is using greedy search to translate a sentence. Why might this strategy fail to find the sequence with the highest overall probability?

24 In beam search text generation with a beam width of , what exactly does the algorithm track at time step ?

25 Why is a length penalty often applied to the scores of candidate sequences during beam search?

26 A text generation model applies Top-K sampling with . If the model predicts a very confident distribution where the top 3 tokens contain of the probability mass, what is a potential drawback of this Top-K approach?

27 In nucleus (Top-p) sampling with , how does the algorithm determine which tokens to keep in the sampling pool?

28 Which of the following describes a scenario where nucleus sampling (Top-p) behaves significantly differently than Top-K sampling?

29 What is the primary objective of instruction tuning a Large Language Model compared to standard pre-training?

30 When performing few-shot prompting with an LLM, the model successfully adapts to a new task. What mechanism allows this adaptation without updating the model's parameters?

31 When utilizing a generative LLM for abstractive summarization, which of the following is the most significant risk compared to extractive summarization?

32 In a multi-turn dialogue generation system, how does an autoregressive LLM typically 'remember' previous conversational turns?

33 How does 'Chain-of-Thought' (CoT) prompting improve an LLM's performance on complex mathematical reasoning tasks?

34 Which of the following best describes the primary difference between how BLEU and ROUGE evaluate generated text?

35 A student evaluates an LLM's response using BLEU and gets a very low score, yet human evaluators rate the response as excellent. What is the most likely reason for this discrepancy?

36 Given a sequence of words , perplexity (PP) is defined as . What does a lower perplexity score on a test set indicate about a language model?

37 Why is perplexity generally considered insufficient on its own for evaluating modern instruction-tuned LLMs?

38 In the context of evaluating LLM hallucination in abstractive summarization, what distinguishes an 'intrinsic hallucination' from an 'extrinsic hallucination'?

39 Which of the following techniques is most commonly used to mitigate factual hallucination in LLMs by grounding the model's responses?

40 When attempting to explain the predictions of a Transformer-based LLM, researchers often look at attention weights. What is a widely recognized limitation of using attention weights as an explainability tool?

41 In nucleus sampling (Top-), the model samples from the smallest set of tokens such that the sum of their probabilities is greater than or equal to . If , and the probability distribution over the vocabulary for the next token is , , , and , what will be the effective size of the sampling vocabulary ?

42 Autoregressive models decoded using standard beam search often exhibit a strong bias towards shorter sequences. To mitigate this, a length penalty is introduced to the objective function: . If , how does this objective theoretically alter the sequence scoring compared to standard beam search?

43 During text generation, consider a scenario where the vocabulary distribution is completely uniform across a large vocabulary . If we transition from Top- sampling with to nucleus sampling with , what happens to the size of the restricted vocabulary from which we sample?

44 Instruction-tuned Large Language Models are often refined using Reinforcement Learning from Human Feedback (RLHF). During the PPO (Proximal Policy Optimization) phase, a Kullback-Leibler (KL) divergence penalty is typically added to the reward. What is the primary analytical purpose of this KL penalty?

45 When evaluating an LLM on a reasoning task using the BLEU score, the resulting score is extremely low, yet human evaluation shows the model's reasoning is perfectly accurate. Which of the following best explains this divergence?

46 Perplexity (PPL) is a standard evaluation metric for language models, defined as . However, models with lower perplexity on a validation set do not always generate text that humans judge as higher quality. Which phenomenon best explains this paradox?

47 Chain-of-Thought (CoT) prompting significantly improves performance on reasoning tasks compared to standard prompting. From a computational complexity perspective of standard Transformer-based LLMs, why does generating intermediate reasoning steps increase the model's problem-solving capability?

48 Consider an autoregressive language model generating a sequence using greedy search. The generated text gets stuck in an infinite loop (e.g., 'I went to the store to the store to the store...'). Which mathematical characteristic of the model's learned distribution most directly contributes to this greedy decoding failure?

49 In the context of LLM hallucination, researchers distinguish between 'intrinsic' and 'extrinsic' hallucinations. An LLM generated a summary stating: 'The CEO of OpenAI, Sam Altman, announced a new model in Paris.' If the source text mentioned the announcement but did not state the location, how is this hallucination classified and why is it notoriously difficult to penalize using standard cross-entropy training?

50 A persistent issue in dialogue generation is 'exposure bias'. Which of the following training paradigms is specifically designed to mitigate exposure bias by bridging the gap between training and inference distributions?

51 In the context of Large Language Models, 'emergent abilities' are capabilities that are not present in smaller models but suddenly appear when the model scale reaches a certain threshold. Which of the following provides the most statistically rigorous critique of emergent abilities as proposed by some recent NLP literature?

52 Contrastive Search is a text generation strategy introduced to prevent degeneration in LLMs. Its objective function at step is formulated to select a token that maximizes model confidence while minimizing what specific component?

53 When leveraging LLMs for abstractive summarization, a phenomenon known as 'lead bias' is frequently observed. If an LLM is heavily exhibiting lead bias, how will this manifest in its outputs, and how does the self-attention mechanism theoretically contribute to it in long documents?

54 A known empirical issue with beam search in neural machine translation and other generative tasks is the 'beam search curse', where increasing the beam size beyond a certain point (e.g., ) degrades the BLEU score. What is the primary cause of this degradation?

55 Which of the following describes a key architectural difference between Causal Language Models (like GPT-3) and Masked Language Models (like BERT) that fundamentally makes Causal LMs better suited for zero-shot generative prompting?

56 To explain the output of a generative LLM, researchers often use Integrated Gradients. For a specific generated token , Integrated Gradients computes the path integral of gradients from a baseline input to the actual input . Why is computing Integrated Gradients for generative LLMs significantly more complex than for standard classification models?

57 In 'Self-Consistency' decoding for LLM reasoning tasks, multiple distinct reasoning paths are sampled, and the final answer is selected via majority vote. For self-consistency to be effective, which decoding strategy MUST be utilized during the generation of the paths?

58 During the creation of an instruction-tuned model, Supervised Fine-Tuning (SFT) is typically performed before RLHF. If the SFT phase trains the model exclusively on examples formatted as User: [Query] Assistant: [Response], and a user at inference prompts the model with System: [Directive], what failure mode is most likely to occur and why?

59 ROUGE-L uses the Longest Common Subsequence (LCS) to evaluate summarization. Let the reference be and the generation be . The length of LCS is 3 (). What is a key mathematical advantage of LCS in ROUGE-L over using purely contiguous n-gram overlaps (like ROUGE-2)?

60 A massive LLM is deployed using pipeline parallelism across multiple GPUs. If the model exhibits high latency during autoregressive token generation (decoding phase) compared to the prefill phase (processing the prompt), what is the primary architectural bottleneck causing this difference?