Back to Subject

Unit 5: Q-Learning & Deep Q- Networks - Practice Quiz

INT423 — Machine Learning-Ii 50 Questions
0 Correct 0 Wrong 50 Left
0/50

1 What type of Reinforcement Learning algorithm is Q-Learning?

A. Model-free, Off-policy
B. Model-based, On-policy
C. Model-free, On-policy
D. Model-based, Off-policy

2 In Q-Learning, what does the 'Q' specifically represent?

A. Quality
B. Quantity
C. Queue
D. Query

3 What is the primary data structure used in basic tabular Q-Learning?

A. A Graph
B. A Q-Table
C. A Decision Tree
D. A Neural Network

4 Which equation is used to update the Q-values in Q-Learning?

A. Euler's Equation
B. Maxwell's Equation
C. Schrodinger Equation
D. Bellman Equation

5 In the Q-learning update rule, what is the role of the learning rate (alpha)?

A. It calculates the total cumulative reward.
B. It controls how much the new information overrides the old information.
C. It determines the probability of exploring a random action.
D. It determines the importance of future rewards.

6 What is the purpose of the discount factor (gamma) in Q-Learning?

A. To set the learning speed
B. To determine the exploration rate
C. To initialize the Q-table
D. To balance immediate and future rewards

7 If the discount factor (gamma) is set to 0, what will the agent optimize for?

A. Long-term cumulative reward
B. Random rewards
C. Only the immediate reward
D. The average reward over time

8 What is the 'Temporal Difference (TD) Error' in the context of Q-Learning?

A. The difference between the target Q-value and the current predicted Q-value
B. The difference between the current Q-value and the previous Q-value
C. The error in the reward function
D. The time it takes to converge

9 What is the Epsilon-Greedy strategy used for?

A. To update the weights of the network
B. To balance exploration and exploitation
C. To calculate the loss function
D. To store experiences in replay memory

10 In an Epsilon-Greedy strategy, what happens if epsilon is 1?

A. The agent always chooses a random action.
B. The agent alternates between best and random actions.
C. The agent stops learning.
D. The agent always chooses the action with the highest Q-value.

11 What is the typical behavior of epsilon during the training process in Deep Q-Learning?

A. It starts low and increases over time.
B. It starts high and decays over time.
C. It remains constant throughout training.
D. It fluctuates randomly.

12 Why does tabular Q-Learning fail in environments like Atari games or Robotics?

A. The rewards are not defined.
B. The state space is too large (Curse of Dimensionality).
C. Q-learning cannot handle discrete actions.
D. The math does not apply to games.

13 What replaces the Q-Table in a Deep Q-Network (DQN)?

A. A Linear Regression model
B. A larger Q-Table
C. A Genetic Algorithm
D. A Deep Neural Network

14 What is the input to the neural network in a standard DQN for playing video games?

A. The Q-value
B. The current score
C. The raw pixels of the game screen (state)
D. The action to be taken

15 What is the output layer size of a DQN used for an environment with 'N' discrete actions?

A. 1 (The best action)
B. 1 (The value of the state)
C. N x N
D. N (One Q-value for each action)

16 What is 'Experience Replay' in DQN?

A. Running the same episode multiple times.
B. Replaying the game after winning.
C. Storing past transitions (s, a, r, s') in a buffer and sampling minibatches for training.
D. Using the target network to replay actions.

17 What is the primary benefit of using Experience Replay?

A. It breaks the correlation between consecutive samples and stabilizes training.
B. It removes the need for a target network.
C. It guarantees finding the global minimum.
D. It increases the epsilon value.

18 In the context of DQN, what is the 'Target Network'?

A. A copy of the main network with frozen weights used to calculate target Q-values.
B. The network that predicts the reward.
C. The network used during the testing phase only.
D. The network that selects the action.

19 Why is a Target Network necessary in DQN?

A. To handle continuous action spaces.
B. To prevent the 'chasing your own tail' instability where target values shift constantly.
C. To speed up the backpropagation process.
D. To increase the exploration rate.

20 How are weights usually updated in the Target Network?

A. Copied from the main network every fixed number of steps.
B. Updated using a different loss function.
C. Continuous backpropagation along with the main network.
D. Randomly initialized every step.

21 What is the loss function typically used in DQN?

A. Kullback-Leibler Divergence
B. Mean Squared Error (MSE) between predicted Q and target Q
C. Cross-Entropy Loss
D. Hinge Loss

22 What main issue does 'Double DQN' address?

A. High memory usage
B. Slow convergence speed
C. Underestimation of Q-values
D. Overestimation of Q-values

23 In standard DQN, how is the target value calculated (ignoring reward and gamma)?

A. Minimum of Q-values
B. Q(s', argmax_a Q(s', a; main_weights); target_weights)
C. max_a Q(s', a; target_weights)
D. Average of Q-values

24 How does Double DQN calculate the target Q-value?

A. It doubles the reward.
B. It uses the Main network to select the best action and the Target network to evaluate its value.
C. It uses the Target network to select the action and the Main network to evaluate it.
D. It uses two totally independent networks trained on different data.

25 What is the architectural change in Dueling DQN compared to standard DQN?

A. It uses Recurrent Neural Networks.
B. It splits the network into two streams: Value stream and Advantage stream.
C. It removes the convolutional layers.
D. It uses two separate neural networks for two different agents.

26 In Dueling DQN, what does the Value function V(s) estimate?

A. How good it is to be in a particular state, regardless of the action taken.
B. The total error.
C. How good a specific action is compared to others.
D. The immediate reward.

27 In Dueling DQN, what does the Advantage function A(s, a) estimate?

A. How much better taking action 'a' is compared to the average action in state 's'.
B. The value of the state.
C. The importance of the state.
D. The probability of winning.

28 How are the Value and Advantage streams combined in Dueling DQN to get Q-values?

A. Concatenation
B. Convolution
C. Aggregation (Summation with normalization)
D. Multiplication

29 What is the main benefit of Dueling DQN?

A. It is computationally cheaper than standard DQN.
B. It works without rewards.
C. It eliminates the need for Experience Replay.
D. It allows the agent to learn which states are valuable without having to learn the effect of every action.

30 Which of the following is true about 'Off-policy' learning in Q-learning?

A. The agent learns the value of the optimal policy regardless of the current actions taken.
B. The agent learns the value of the policy it is currently executing.
C. It cannot use Experience Replay.
D. It requires a model of the environment.

31 What represents the 'State' in a Reinforcement Learning framework?

A. The decision maker.
B. The current situation or configuration of the environment.
C. The feedback from the environment.
D. The move made by the agent.

32 In the Bellman optimality equation, what does 'max_a Q(s', a)' represent?

A. The immediate reward.
B. The value of the best action available in the next state.
C. The worst possible future outcome.
D. The average value of the next state.

33 If an agent reaches a terminal state, what is the Target Q-value?

A. The immediate reward (r) only.
B. Reward + gamma * max Q.
C. Zero.
D. Infinity.

34 Which of the following creates a 'Moving Target' problem in naive Deep Q-Learning?

A. Using a fixed target network.
B. Using a small learning rate.
C. Using a replay buffer.
D. Using the same network to calculate both predicted value and target value.

35 What is 'Catastrophic Forgetting' in the context of RL?

A. The gradients vanish.
B. The agent forgets previously learned knowledge when training on new dissimilar experiences.
C. The replay buffer gets deleted.
D. The agent forgets the goal.

36 In Q-Learning, convergence to the optimal Q-values is guaranteed if:

A. All state-action pairs are visited infinitely often and learning rate decays appropriately.
B. The discount factor is 1.
C. Epsilon is kept at 1.0.
D. The neural network is deep enough.

37 Which component of the tuple (S, A, R, S') is NOT known before the agent takes an action?

A. None of the above
B. A (Action chosen)
C. S (Current State)
D. R and S' (Reward and Next State)

38 In Double DQN, the update equation replaces the target 'Y' with:

A. R + gamma * Q_main(s', argmax Q_target(s', a))
B. R + gamma * max Q_target(s', a)
C. R + gamma * Q_target(s', argmax Q_main(s', a))
D. R + gamma * V(s')

39 What is the primary motivation for 'Prioritized Experience Replay'?

A. To ensure random sampling.
B. To replay experiences where the agent had a high TD error (learned the most).
C. To replay recent experiences first.
D. To save memory.

40 When preprocessing images for DQN (e.g., Atari), what is a common technique?

A. Inverting colors.
B. Increasing resolution to 4K.
C. Converting to grayscale and resizing.
D. Adding noise.

41 In Dueling DQN, the aggregation layer usually subtracts the mean of the Advantage values. Why?

A. It is a requirement of the activation function.
B. To make the values positive.
C. To reduce the size of the output.
D. For numerical stability and identifiability.

42 What is an 'Episode' in Reinforcement Learning?

A. The entire training process.
B. A single update of the Q-table.
C. A sequence of states, actions, and rewards from start to a terminal state.
D. One step of training.

43 Which activation function is commonly used in the hidden layers of a Deep Q-Network?

A. Softmax
B. Sigmoid
C. Step function
D. ReLU (Rectified Linear Unit)

44 Why is the Softmax function generally NOT used in the output layer of a DQN?

A. DQN outputs Q-values (regression), not probabilities (classification).
B. It is not differentiable.
C. It cannot handle negative numbers.
D. It is too slow.

45 In the context of RL, what is 'Exploitation'?

A. Selecting the action currently believed to be optimal.
B. Increasing the discount factor.
C. Stopping the training early.
D. Trying new actions to gather information.

46 What is 'Frame Stacking' in DQN for Atari games?

A. Stacking Q-tables on top of each other.
B. Stacking multiple neural networks.
C. Stacking rewards.
D. Stacking consecutive frames to capture motion/velocity.

47 What optimization algorithm is typically used to train the DQN weights?

A. K-Means Clustering
B. Principal Component Analysis
C. Gradient Descent (e.g., RMSProp or Adam)
D. Genetic Algorithms

48 If the Q-values for all actions in a state are equal, what will an epsilon-greedy policy (with epsilon=0) do?

A. Choose an action randomly among them (or the first one).
B. Stop the episode.
C. Choose no action.
D. Increase epsilon.

49 Which of the following implies that an RL problem is 'episodic'?

A. The environment is deterministic.
B. The agent runs forever.
C. The task breaks down into independent sequences ending in a terminal state.
D. The discount factor is 1.

50 What is the primary reason DQN was considered a breakthrough (published by DeepMind)?

A. It was the first algorithm to master a wide range of Atari 2600 games using only raw pixels and scores.
B. It solved the traveling salesman problem.
C. It used a new type of CPU.
D. It proved that gamma should always be 0.99.
Previous MCQ
Unit 4
Next MCQ
Unit 6