Unit 5 - Practice Quiz

INT423 50 Questions
0 Correct 0 Wrong 50 Left
0/50

1 What type of Reinforcement Learning algorithm is Q-Learning?

A. Model-based, Off-policy
B. Model-free, On-policy
C. Model-based, On-policy
D. Model-free, Off-policy

2 In Q-Learning, what does the 'Q' specifically represent?

A. Quantity
B. Query
C. Queue
D. Quality

3 What is the primary data structure used in basic tabular Q-Learning?

A. A Graph
B. A Q-Table
C. A Neural Network
D. A Decision Tree

4 Which equation is used to update the Q-values in Q-Learning?

A. Schrodinger Equation
B. Bellman Equation
C. Maxwell's Equation
D. Euler's Equation

5 In the Q-learning update rule, what is the role of the learning rate (alpha)?

A. It calculates the total cumulative reward.
B. It determines the probability of exploring a random action.
C. It controls how much the new information overrides the old information.
D. It determines the importance of future rewards.

6 What is the purpose of the discount factor (gamma) in Q-Learning?

A. To determine the exploration rate
B. To set the learning speed
C. To balance immediate and future rewards
D. To initialize the Q-table

7 If the discount factor (gamma) is set to 0, what will the agent optimize for?

A. Only the immediate reward
B. The average reward over time
C. Long-term cumulative reward
D. Random rewards

8 What is the 'Temporal Difference (TD) Error' in the context of Q-Learning?

A. The difference between the target Q-value and the current predicted Q-value
B. The difference between the current Q-value and the previous Q-value
C. The error in the reward function
D. The time it takes to converge

9 What is the Epsilon-Greedy strategy used for?

A. To store experiences in replay memory
B. To update the weights of the network
C. To calculate the loss function
D. To balance exploration and exploitation

10 In an Epsilon-Greedy strategy, what happens if epsilon is 1?

A. The agent alternates between best and random actions.
B. The agent stops learning.
C. The agent always chooses a random action.
D. The agent always chooses the action with the highest Q-value.

11 What is the typical behavior of epsilon during the training process in Deep Q-Learning?

A. It starts low and increases over time.
B. It starts high and decays over time.
C. It remains constant throughout training.
D. It fluctuates randomly.

12 Why does tabular Q-Learning fail in environments like Atari games or Robotics?

A. The state space is too large (Curse of Dimensionality).
B. The rewards are not defined.
C. The math does not apply to games.
D. Q-learning cannot handle discrete actions.

13 What replaces the Q-Table in a Deep Q-Network (DQN)?

A. A larger Q-Table
B. A Genetic Algorithm
C. A Linear Regression model
D. A Deep Neural Network

14 What is the input to the neural network in a standard DQN for playing video games?

A. The action to be taken
B. The raw pixels of the game screen (state)
C. The current score
D. The Q-value

15 What is the output layer size of a DQN used for an environment with 'N' discrete actions?

A. N (One Q-value for each action)
B. 1 (The best action)
C. 1 (The value of the state)
D. N x N

16 What is 'Experience Replay' in DQN?

A. Running the same episode multiple times.
B. Replaying the game after winning.
C. Using the target network to replay actions.
D. Storing past transitions (s, a, r, s') in a buffer and sampling minibatches for training.

17 What is the primary benefit of using Experience Replay?

A. It guarantees finding the global minimum.
B. It breaks the correlation between consecutive samples and stabilizes training.
C. It increases the epsilon value.
D. It removes the need for a target network.

18 In the context of DQN, what is the 'Target Network'?

A. The network used during the testing phase only.
B. A copy of the main network with frozen weights used to calculate target Q-values.
C. The network that predicts the reward.
D. The network that selects the action.

19 Why is a Target Network necessary in DQN?

A. To prevent the 'chasing your own tail' instability where target values shift constantly.
B. To speed up the backpropagation process.
C. To increase the exploration rate.
D. To handle continuous action spaces.

20 How are weights usually updated in the Target Network?

A. Updated using a different loss function.
B. Copied from the main network every fixed number of steps.
C. Randomly initialized every step.
D. Continuous backpropagation along with the main network.

21 What is the loss function typically used in DQN?

A. Mean Squared Error (MSE) between predicted Q and target Q
B. Kullback-Leibler Divergence
C. Cross-Entropy Loss
D. Hinge Loss

22 What main issue does 'Double DQN' address?

A. High memory usage
B. Underestimation of Q-values
C. Slow convergence speed
D. Overestimation of Q-values

23 In standard DQN, how is the target value calculated (ignoring reward and gamma)?

A. max_a Q(s', a; target_weights)
B. Average of Q-values
C. Q(s', argmax_a Q(s', a; main_weights); target_weights)
D. Minimum of Q-values

24 How does Double DQN calculate the target Q-value?

A. It uses the Main network to select the best action and the Target network to evaluate its value.
B. It uses two totally independent networks trained on different data.
C. It uses the Target network to select the action and the Main network to evaluate it.
D. It doubles the reward.

25 What is the architectural change in Dueling DQN compared to standard DQN?

A. It uses Recurrent Neural Networks.
B. It uses two separate neural networks for two different agents.
C. It splits the network into two streams: Value stream and Advantage stream.
D. It removes the convolutional layers.

26 In Dueling DQN, what does the Value function V(s) estimate?

A. The total error.
B. How good a specific action is compared to others.
C. How good it is to be in a particular state, regardless of the action taken.
D. The immediate reward.

27 In Dueling DQN, what does the Advantage function A(s, a) estimate?

A. How much better taking action 'a' is compared to the average action in state 's'.
B. The value of the state.
C. The importance of the state.
D. The probability of winning.

28 How are the Value and Advantage streams combined in Dueling DQN to get Q-values?

A. Multiplication
B. Aggregation (Summation with normalization)
C. Concatenation
D. Convolution

29 What is the main benefit of Dueling DQN?

A. It works without rewards.
B. It is computationally cheaper than standard DQN.
C. It eliminates the need for Experience Replay.
D. It allows the agent to learn which states are valuable without having to learn the effect of every action.

30 Which of the following is true about 'Off-policy' learning in Q-learning?

A. The agent learns the value of the optimal policy regardless of the current actions taken.
B. The agent learns the value of the policy it is currently executing.
C. It cannot use Experience Replay.
D. It requires a model of the environment.

31 What represents the 'State' in a Reinforcement Learning framework?

A. The feedback from the environment.
B. The decision maker.
C. The current situation or configuration of the environment.
D. The move made by the agent.

32 In the Bellman optimality equation, what does 'max_a Q(s', a)' represent?

A. The worst possible future outcome.
B. The value of the best action available in the next state.
C. The immediate reward.
D. The average value of the next state.

33 If an agent reaches a terminal state, what is the Target Q-value?

A. Reward + gamma * max Q.
B. The immediate reward (r) only.
C. Zero.
D. Infinity.

34 Which of the following creates a 'Moving Target' problem in naive Deep Q-Learning?

A. Using a small learning rate.
B. Using a replay buffer.
C. Using the same network to calculate both predicted value and target value.
D. Using a fixed target network.

35 What is 'Catastrophic Forgetting' in the context of RL?

A. The agent forgets previously learned knowledge when training on new dissimilar experiences.
B. The gradients vanish.
C. The replay buffer gets deleted.
D. The agent forgets the goal.

36 In Q-Learning, convergence to the optimal Q-values is guaranteed if:

A. Epsilon is kept at 1.0.
B. The discount factor is 1.
C. The neural network is deep enough.
D. All state-action pairs are visited infinitely often and learning rate decays appropriately.

37 Which component of the tuple (S, A, R, S') is NOT known before the agent takes an action?

A. S (Current State)
B. R and S' (Reward and Next State)
C. None of the above
D. A (Action chosen)

38 In Double DQN, the update equation replaces the target 'Y' with:

A. R + gamma * Q_main(s', argmax Q_target(s', a))
B. R + gamma * max Q_target(s', a)
C. R + gamma * Q_target(s', argmax Q_main(s', a))
D. R + gamma * V(s')

39 What is the primary motivation for 'Prioritized Experience Replay'?

A. To save memory.
B. To replay experiences where the agent had a high TD error (learned the most).
C. To replay recent experiences first.
D. To ensure random sampling.

40 When preprocessing images for DQN (e.g., Atari), what is a common technique?

A. Converting to grayscale and resizing.
B. Increasing resolution to 4K.
C. Adding noise.
D. Inverting colors.

41 In Dueling DQN, the aggregation layer usually subtracts the mean of the Advantage values. Why?

A. For numerical stability and identifiability.
B. To reduce the size of the output.
C. It is a requirement of the activation function.
D. To make the values positive.

42 What is an 'Episode' in Reinforcement Learning?

A. A sequence of states, actions, and rewards from start to a terminal state.
B. One step of training.
C. A single update of the Q-table.
D. The entire training process.

43 Which activation function is commonly used in the hidden layers of a Deep Q-Network?

A. Sigmoid
B. ReLU (Rectified Linear Unit)
C. Step function
D. Softmax

44 Why is the Softmax function generally NOT used in the output layer of a DQN?

A. It cannot handle negative numbers.
B. It is too slow.
C. It is not differentiable.
D. DQN outputs Q-values (regression), not probabilities (classification).

45 In the context of RL, what is 'Exploitation'?

A. Selecting the action currently believed to be optimal.
B. Stopping the training early.
C. Increasing the discount factor.
D. Trying new actions to gather information.

46 What is 'Frame Stacking' in DQN for Atari games?

A. Stacking consecutive frames to capture motion/velocity.
B. Stacking Q-tables on top of each other.
C. Stacking rewards.
D. Stacking multiple neural networks.

47 What optimization algorithm is typically used to train the DQN weights?

A. Principal Component Analysis
B. Gradient Descent (e.g., RMSProp or Adam)
C. K-Means Clustering
D. Genetic Algorithms

48 If the Q-values for all actions in a state are equal, what will an epsilon-greedy policy (with epsilon=0) do?

A. Choose an action randomly among them (or the first one).
B. Stop the episode.
C. Choose no action.
D. Increase epsilon.

49 Which of the following implies that an RL problem is 'episodic'?

A. The task breaks down into independent sequences ending in a terminal state.
B. The environment is deterministic.
C. The discount factor is 1.
D. The agent runs forever.

50 What is the primary reason DQN was considered a breakthrough (published by DeepMind)?

A. It solved the traveling salesman problem.
B. It was the first algorithm to master a wide range of Atari 2600 games using only raw pixels and scores.
C. It used a new type of CPU.
D. It proved that gamma should always be 0.99.