Unit 3 - Practice Quiz

Correct Answer: It is a Lazy learning algorithm that stores the dataset and performs computation only during prediction.

Explanation: KNN is a lazy learner (instance-based learning) because it does not learn a discriminative function from the training data but memorizes the training dataset instead.

Correct Answer: The model captures noise in the training data, leading to overfitting.

Explanation: With a very small (like ), the algorithm is highly sensitive to noise and local outliers, resulting in a complex decision boundary and overfitting (high variance).

Correct Answer: Euclidean Distance

Explanation: The formula represents the Euclidean Distance, which is the straight-line distance between two points in Euclidean space.

Correct Answer: Because KNN is based on distance metrics, and features with larger scales will dominate the distance calculation.

Explanation: If one feature has a range of 0-1000 and another 0-1, the distance calculation will be heavily biased toward the first feature. Scaling ensures all features contribute equally.

Correct Answer: Prediction time becomes very slow because it scans the entire dataset.

Explanation: Since KNN is a lazy learner, the computational cost happens at the inference (prediction) stage. For every new query, it must calculate distances to all training points, making it slow for large datasets.

Correct Answer: Use Cross-Validation (e.g., Elbow method) to minimize error.

Explanation: Cross-validation involves testing different values and selecting the one that minimizes the validation error rate to balance bias and variance.

Correct Answer: A class label or a continuous value.

Explanation: Leaf nodes (terminal nodes) represent the final outcome, which is a class label (in classification) or a numerical value (in regression).

Correct Answer: They make the locally optimal choice at each step with the hope of finding a global optimum.

Explanation: A greedy algorithm makes the best immediate decision at the current node (e.g., best split) without considering future consequences, often leading to a suboptimal global tree but offering computational efficiency.

Correct Answer: Information Gain

Explanation: ID3 uses Information Gain, which relies on the reduction of Entropy, to determine the best attribute to split the data.

Correct Answer: $0$

Explanation: Entropy measures impurity. If all examples are of the same class, there is no impurity/uncertainty, so Entropy is 0.

Correct Answer: $1$

Explanation: Using the formula , calculation: . This represents maximum impurity.

Correct Answer: It is biased towards attributes with a large number of distinct values.

Explanation: Information Gain favors attributes like 'ID' or 'Date' which have unique values for every instance, resulting in pure but useless splits. This bias is corrected by Gain Ratio.

Correct Answer: C4.5

Explanation: The C4.5 algorithm introduced Gain Ratio, which normalizes the Information Gain by the 'Split Info' (intrinsic information) to penalize attributes with many branches.

Correct Answer: Gini Index

Explanation: CART uses the Gini Index (or Gini Impurity) to measure the likelihood of incorrect classification of a new instance if it were randomly classified according to the distribution of class labels.

Correct Answer:

Explanation: For binary classification, Gini is . The minimum is $0$ (pure) and the maximum is $0.5$ (perfectly mixed, ; ).

Correct Answer: It produces only binary splits (two child nodes).

Explanation: CART constructs binary trees. Even if a categorical attribute has multiple values, CART groups them into two subsets for the split.

Correct Answer: To reduce the complexity of the tree and prevent overfitting.

Explanation: Pruning removes sections of the tree that provide little power to classify instances, thereby reducing complexity and improving generalization (preventing overfitting).

Correct Answer: Halt the construction of the tree early if goodness measures fall below a threshold.

Explanation: Pre-pruning stops the tree generation process before it fully classifies the training set (e.g., max depth reached, min samples per split), whereas post-pruning cuts back a fully grown tree.

Correct Answer: Cost Complexity Pruning (Weakest Link Pruning)

Explanation: Cost Complexity Pruning is a post-pruning method that adds a penalty for the number of terminal nodes to the error rate, removing subtrees that do not justify their complexity.

Correct Answer: Distributing the instance to all child nodes with weights proportional to the population of the child nodes.

Explanation: C4.5 handles missing attribute values by sending the instance down all branches, weighted by the probability of seeing that branch's value in the training data.

Correct Answer: Low training error, high testing error.

Explanation: Overfitting occurs when the model memorizes the training data (low training error) but fails to generalize to unseen data (high testing error).

Correct Answer: To combine multiple weak models to create a strong predictive model.

Explanation: Ensemble learning aggregates the predictions of multiple base estimators (weak learners) to improve generalizability and robustness over a single estimator.

Correct Answer: Bias and Variance

Explanation: Ensembles aim to lower the Bias (error from erroneous assumptions) and Variance (sensitivity to small fluctuations in the training set).

Correct Answer: Bagging

Explanation: Bagging stands for Bootstrap Aggregating. It involves training models on random subsets of the data with replacement.

Correct Answer: By sampling with replacement from the original dataset.

Explanation: Bagging uses sampling with replacement, meaning the same data point can appear multiple times in a single training subset.

Correct Answer: It selects a random subset of features for splitting at each node.

Explanation: Random Forest decorrelates the trees by considering only a random subset of features at each split, preventing a single dominant feature from dictating the structure of all trees.

Correct Answer: The error calculated on the data samples that were not included in the bootstrap sample for a specific tree.

Explanation: Since Bagging uses sampling with replacement, some data is left out for each tree. These OOB samples act as an internal validation set to estimate prediction error.

Correct Answer: Models are trained sequentially, with each correcting the errors of the predecessor.

Explanation: Boosting is a sequential process where subsequent models focus on the instances that were misclassified by previous models.

Correct Answer: Weights of misclassified instances are increased.

Explanation: AdaBoost increases the weights of misclassified instances so that the next weak learner focuses more on getting these difficult cases right.

Correct Answer: Bagging reduces variance; Boosting primarily reduces bias.

Explanation: Bagging (like Random Forest) averages high-variance models to reduce variance. Boosting takes high-bias weak learners and improves them sequentially to reduce bias.

Correct Answer: Optimizes a loss function by training new models on the residual errors of previous models.

Explanation: Gradient Boosting uses a gradient descent approach. Instead of updating weights, it trains the next model to predict the residuals (errors) of the current ensemble.

Correct Answer: Training a meta-model to learn how to combine the predictions of base models.

Explanation: Stacking involves training base models and then training a meta-learner (or blender) that takes the predictions of the base models as input features to make the final prediction.

Correct Answer: Selecting the class with the majority of votes from the classifiers.

Explanation: Hard Voting predicts the final class label based on the mode (majority vote) of the class labels predicted by the individual classifiers.

Correct Answer: Averaging the predicted class probabilities and choosing the class with the highest average probability.

Explanation: Soft Voting considers the confidence of the classifiers by averaging their probability outputs, often yielding better results than hard voting if the classifiers are well-calibrated.

Correct Answer: XGBoost

Explanation: XGBoost (eXtreme Gradient Boosting) is a highly efficient and scalable implementation of gradient boosting.

Correct Answer: In high-dimensional space, data becomes sparse, and all points tend to be equidistant.

Explanation: As dimensions increase, the volume of the space increases exponentially, making data sparse. Distance metrics lose meaning as the ratio of the distance to the nearest neighbor vs. the farthest neighbor approaches 1.

Correct Answer: Sum of Squared Errors (Variance reduction)

Explanation: For regression trees, the goal is to minimize the variance within the nodes. This is usually measured by the Sum of Squared Errors (SSE) or Mean Squared Error.

Correct Answer: Yes, they can handle both.

Explanation: Decision Trees are versatile and can natively handle both numerical and categorical attributes (though implementation details vary between ID3, C4.5, and CART).

Correct Answer: A model that performs slightly better than random guessing.

Explanation: In Boosting theory, a weak learner is a classifier that is only required to be slightly better than random chance (accuracy > 0.5 for binary classification).

Correct Answer:

Explanation: Information Gain is the entropy of the parent node minus the weighted sum of the entropy of the child nodes resulting from the split.

Correct Answer: Logistic Regression

Explanation: Logistic Regression is a standalone parametric linear classification model, not an ensemble method.

Correct Answer: It reduces the risk of overfitting by averaging multiple trees.

Explanation: A single deep decision tree has high variance (overfitting). Random Forest averages many uncorrelated trees, effectively reducing variance and improving generalization.

Correct Answer: It scales the contribution of each tree to the final prediction.

Explanation: The learning rate (shrinkage) shrinks the contribution of each new tree added to the model. A lower learning rate requires more trees but usually leads to better generalization.

Correct Answer: Models are assigned weights based on their performance (e.g., accuracy), and the weighted sum determines the class.

Explanation: In Weighted Voting, better-performing models are trusted more. Their votes are multiplied by a weight derived from their validation performance.

Correct Answer: Manhattan Distance

Explanation: Manhattan Distance is the sum of absolute differences (), which corresponds to the norm.

Correct Answer: High Variance (Overfitting)

Explanation: A fully grown tree captures every distinct pattern and noise in the training data, leading to high variance and overfitting.

Correct Answer: AdaBoost

Explanation: AdaBoost typically uses Decision Stumps—trees with only one split—as the weak learners.

Correct Answer: Averaging is for regression; Voting is for classification.

Explanation: Generally, Voting (counting discrete class labels) is used for classification tasks, while Averaging (taking the mean of numerical predictions) is used for regression tasks.

Correct Answer:

Explanation: The Gini Index is calculated by subtracting the sum of the squared probabilities of each class from one.

Correct Answer: It learns the optimal combination of base models rather than assuming equal or fixed weights.

Explanation: Stacking uses a meta-learner (second-level model) to learn the best way to combine predictions, effectively correcting biases that simple voting or averaging might miss.