Unit 6: Generative NLP and LLMs - Practice Quiz

1 What is the main purpose of a generative NLP model?

2 During text generation, what does an autoregressive language model usually predict?

3 Why are text generation strategies used with language models?

4 How does greedy search select the next token?

5 What does beam search keep during text generation?

6 What does the beam width control in beam search?

7 In top-k sampling, which tokens can be sampled at each step?

8 How does nucleus sampling choose its candidate token set?

9 What are instruction-tuned large language models trained to do?

10 Which prompt is most directly suited to an instruction-tuned model?

11 What should a good text summary mainly preserve?

12 Which summarization behavior is undesirable?

13 What is an important quality of a dialogue generation system?

14 Which activity is an example of a reasoning task for a language model?

15 What is the purpose of an evaluation metric in NLP?

16 What does lower perplexity generally indicate for a language model on a test set?

17 Perplexity is most directly based on which model output?

18 Which quality can human evaluators directly rate in generated text?

19 What is the main goal of explainability in large language models?

20 What is a hallucination in the output of a large language model?

21 A sequence-to-sequence model performs well during training with teacher forcing but produces increasingly poor text at inference time after making one early mistake. Which issue best explains this behavior?

22 A developer wants a language model to produce several varied story ideas from the same prompt while avoiding extremely unlikely tokens. Which decoding setup is most appropriate?

23 At the first decoding step, token A has probability and token B has probability . Their best possible next-token probabilities are and , respectively. Which sequence does greedy search select based on these values?

24 A beam-search system consistently favors very short translations because sequence probabilities shrink as more token probabilities are multiplied. Which modification most directly addresses this problem?

25 A model uses top-k sampling with . At a particular decoding step, the five retained tokens together contain only of the original probability mass. What happens before sampling?

26 A model uses nucleus sampling with threshold . Why can the number of candidate tokens change from one decoding step to another?

27 A pretrained language model often continues a user's prompt instead of carrying out the requested task. Why can instruction tuning improve this behavior?

28 An instruction-tuned model answers common tasks well but performs poorly when a request uses an unfamiliar format. Which intervention is most likely to improve its performance on that format?

29 An abstractive summarizer produces fluent summaries but occasionally changes dates and names from the source document. Which evaluation should be prioritized?

30 A dialogue model gives a restaurant recommendation and later denies making that recommendation. Which capability most directly needs improvement?

31 A model solves single-operation arithmetic problems but often fails on problems requiring several dependent operations. Which prompting method is most likely to improve accuracy?

32 Two machine-generated summaries express the same meaning as a reference summary but use different wording. Their ROUGE scores are unexpectedly low. What is the most likely reason?

33 A language model assigns probabilities and to the two correct tokens in a sequence. Using , what is the sequence perplexity?

34 Model X reports lower perplexity than Model Y, but the models use different tokenizers. Why should the comparison be treated cautiously?

35 Three annotators rate dialogue responses for relevance on a five-point scale, but their scores differ substantially. Which measure would best assess rating consistency?

36 A team compares two dialogue models through human evaluation. Which design best reduces bias caused by model identity and response order?

37 An analyst claims that the token receiving the highest attention weight is always the sole cause of a model's prediction. What is the main problem with this claim?

38 To test whether a phrase truly influences a classifier's output, an analyst removes the phrase and observes the change in prediction. What explainability approach is being used?

39 A question-answering model invents recent company statistics that were not present in its training data. Which system change most directly reduces this type of hallucination?

40 A medical summarization system generates a statement that is plausible but absent from the patient's record. How should this output be classified?

41 An autoregressive language model factorizes a sequence probability as . During training, all target positions can still be processed in parallel using teacher forcing. Why does this not permit fully parallel generation?

42 At the first decoding step, a model assigns and . At the next step, and , with all other continuations having lower complete-sequence probability. Which sequence does greedy decoding return, and which has the greatest joint probability?

43 A decoder ranks completed hypotheses by the unnormalized score . Assuming token probabilities are strictly below , what systematic preference can this introduce, and what modification most directly addresses it?

44 A width- beam discards a prefix at step because its current cumulative log probability ranks third. Later, that prefix would have led to the globally highest-scoring completed sequence. Which statement best characterizes this outcome?

45 Let contain the tokens with highest model probabilities. Which distribution is sampled by standard top-k decoding before any temperature adjustment?

46 A model's sorted next-token probabilities are . Under nucleus sampling with threshold , what is the sampling probability of the third token after truncation and renormalization?

47 Two candidate tokens have original probability ratio . Temperature scaling uses . What is the new ratio when ?

48 A developer wants generated answers to be reproducible while retaining more global sequence optimization than greedy decoding. Which configuration best satisfies both goals?

49 An instruction-tuned model follows requested output formats much better than its base model but confidently repeats the same false historical claim. Which explanation is most accurate?

50 A supervised instruction-tuning dataset contains many prompts where uncertain questions are answered with polished guesses rather than abstentions. What behavior is most likely after fine-tuning?

51 A system summary states, "The treatment reduced mortality," while the source says, "The treatment did not reduce mortality." The summary contains no outside information. What is the most precise diagnosis?

52 A dialogue model is coherent for the first few turns but gradually repeats itself and responds to claims that no participant made. Which training-inference mismatch most directly contributes to this pattern?

53 For a reasoning benchmark, a model samples 25 diverse reasoning paths and returns the answer occurring most often. Under what condition can this self-consistency strategy still reliably select a wrong answer?

54 A semantically correct generated answer uses different wording from a single reference and receives a low BLEU score. Which property of BLEU best explains this result?

55 A three-token sequence has token negative log-likelihoods , , and . Using natural logarithms, what is its token-level perplexity?

56 Model X reports lower token-level perplexity than Model Y on the same raw documents, but the models use different tokenizers. Why is the direct comparison potentially invalid?

57 Two summarization systems are compared by asking each annotator to choose the better output for the same document. Which design most directly controls for annotator severity and document difficulty?

58 A researcher presents one attention head's weights as proof that specific input tokens caused an LLM's answer. What is the strongest methodological objection?

59 An explanation method identifies a hidden unit as representing negation. Which experiment provides the strongest evidence that the unit causally affects negation-sensitive predictions?

60 A retrieval-augmented LLM receives a passage containing the correct answer but generates a conflicting claim. Which conclusion is best supported?