Unit 2: Feature Engineering and Dimensionality Reduction - Practice Quiz

Variance Threshold is a simple baseline approach to feature selection. It removes all features whose variance doesn’t meet a certain threshold. A feature with zero variance has the same value for all samples and thus provides no predictive information.

Variance measures how much the values in a feature spread out from their average. A variance of 0 means there is no spread at all, indicating that every sample has the exact same value for that feature, making it a constant.

This method is used to identify and remove features that are highly correlated with one another (multicollinearity). Keeping both features can make models less interpretable and stable, so one of them is often dropped.

Multicollinearity occurs when independent variables in a regression model are highly correlated. This can inflate the variance of the coefficient estimates and make the model unstable. Removing one of the correlated features is a common way to address this.

Forward Selection is an iterative method that begins with no features in the model. In each iteration, it adds the feature that provides the greatest improvement to the model's performance, until no further significant improvement can be made.

Backward Elimination starts with a model containing all candidate features. It then iteratively removes the least useful feature, one at a time, until a stopping criterion is met, such as all remaining features having a certain level of statistical significance.

Tree-based models like Random Forest calculate feature importance by looking at how much each feature contributes to reducing impurity (e.g., Gini impurity or entropy) at the nodes where it is used for splitting. The average decrease across all trees gives its overall importance score.

Feature Extraction involves transforming the original features into a new, smaller set of features (e.g., principal components). This is different from Feature Selection, which only selects a subset of the original features.

Creating new features (also known as feature engineering) involves deriving new information from existing data. Extracting components like the month, year, or day of the week from a date can reveal cyclical patterns that a model can use.

This is a classic example of feature engineering, where domain knowledge is used to create a new, potentially more informative feature (a ratio in this case) from existing ones.

Aggregation features are created by grouping data by a specific attribute (like 'customer_id') and then applying an aggregation function (like sum, mean, or count) to another attribute.

Dimensionality reduction techniques aim to decrease the number of input variables (features or dimensions) while trying to preserve as much of the important information as possible.

By reducing the number of features, dimensionality reduction can significantly speed up the training process and make the model simpler, which can also help in preventing overfitting.

This term describes how data in high-dimensional spaces becomes very sparse. The volume of the space grows so fast that the data points become isolated, making it difficult for algorithms to find meaningful patterns.

PCA is an unsupervised technique because it does not use the target variable. It transforms the data to a new coordinate system to reduce dimensionality by finding the directions (principal components) of maximum variance.

PCA creates new features called principal components which are linear combinations of the original features. A key characteristic is that these new components are orthogonal (uncorrelated) to each other.

The first principal component is defined as the linear combination of the original features that captures the largest possible variance in the dataset. Subsequent components capture the maximum remaining variance while being orthogonal to the previous ones.

The explained variance ratio tells us how much of the information (variance) from the original dataset is contained within each principal component. This helps in deciding how many components to keep.

Unlike PCA which focuses on capturing variance, LDA's primary goal is to find the linear combinations of features that best separate the different classes in a dataset. It aims to maximize the distance between the means of the classes while minimizing the variance within each class.

LDA is a supervised algorithm because it uses the class labels of the data points to find the axes that maximize the separation between those classes. PCA, in contrast, is unsupervised as it only looks at the variance within the features, not the target labels.

The variance threshold method removes all features whose variance does not meet the specified threshold. The threshold is 0.01. The variances are: Var(F1) = 0.02 > 0.01, Var(F2) = 0.15 > 0.01, Var(F3) = 0.005 < 0.01, and Var(F4) = 0.20 > 0.01. Therefore, feature F3 is removed, and features F1, F2, and F4 are kept.

When two features are highly correlated, the standard practice is to keep the one that has a stronger relationship with the target variable. The strength of this relationship is measured by the absolute value of the correlation. Here, and . Since 0.6 > 0.4, feature_B is considered more predictive. Therefore, feature_A should be removed.

Forward selection is a greedy algorithm. It adds the best feature at each step based on the current set of features. However, it never re-evaluates previously added features. This can lead to a final feature set that is suboptimal, as a combination of two features added later might have been better than the first feature chosen, but the algorithm is locked into its initial choice and cannot backtrack.

Backward elimination is an iterative process. It starts with all features and, at each step, removes the single feature whose removal leads to the best model performance. After identifying and removing F7 in the first step, the process continues with the remaining 9 features. The cycle of training models with one feature removed (now from a set of 9) is repeated to find the next feature to eliminate.

Impurity-based feature importance measures have a known bias: they tend to favor features with a higher number of unique values (high cardinality). This applies to both numerical features and categorical features with many levels. These features offer more potential split points, which can lead to a larger reduction in impurity purely by chance, thus inflating their perceived importance.

Feature Extraction involves creating a new, smaller set of features by combining or transforming the original ones. PCA does this by finding linear combinations of the original pixel features to create new, uncorrelated features (principal components). This is distinct from Feature Selection, which would involve keeping a subset of the original 784 pixels and discarding the rest.

While features like day of the week, weekend status, and month are all useful contextual features, departure_difference directly calculates the delay at the point of departure. This is a very strong and direct signal for predicting the total flight delay, as it captures the initial error that the model is trying to predict.

Aggregation features involve summarizing data from a lower level of detail (individual transactions) to a higher level (customer). Calculating the average transaction amount per customer (GROUP BY customer_id) creates a single value for each customer that represents their typical spending behavior. This is a powerful feature for a customer-level prediction task like churn.

When features are highly correlated, they contain redundant information. Feature selection would discard some of these features, losing their unique contribution entirely. PCA, a dimensionality reduction technique, combines the correlated features into a smaller set of new, uncorrelated components. Each component captures a part of the variance from the original set, effectively summarizing the information without completely discarding it.

In high-dimensional spaces, the volume of the space increases exponentially, causing data points to become very sparse. A key consequence is that the distance to the nearest neighbor approaches the distance to the farthest neighbor for most points. This phenomenon, known as distance concentration, makes the concept of a "close" neighbor less distinct and meaningful, which undermines the core assumption of distance-based algorithms like k-NN.

The vector [0.707, -0.707] represents the loadings for PC1. This means PC1 can be written as the linear combination: PC1 = 0.707 * 'study_hours' - 0.707 * 'sleep_hours'. The opposite signs of the loadings indicate that the direction of maximum variance in the data is one where an increase in 'study_hours' corresponds to a decrease in 'sleep_hours', signifying a strong negative correlation.

To find the cumulative variance, we sum the explained variance ratios of the components.

• 1 component: 0.55 (55%)
• 2 components: 0.55 + 0.25 = 0.80 (80%)
• 3 components: 0.55 + 0.25 + 0.12 = 0.92 (92%)
  To capture at least 90% of the variance, a cumulative sum of 0.90 or more is required. This is first achieved with the top 3 components, which together explain 92% of the variance.

This is the core distinction. PCA is an unsupervised technique that finds the directions (principal components) of maximum variance in the dataset, without any regard for class labels. In contrast, LDA is a supervised technique that uses class labels to find a new feature subspace that maximizes the ratio of between-class variance to within-class variance, thereby maximizing class separability.

Variance is highly dependent on the scale of a feature. A feature with a large scale (like 'income') will naturally have a much larger variance than a feature with a small scale (like 'age'), which can be misleading. To compare variances fairly, all features must be brought to a common scale. Feature scaling, such as normalizing to a [0, 1] range or standardizing to have a mean of 0 and standard deviation of 1, is a crucial prerequisite.

Backward elimination begins by building a model with all features. In a high-dimensional dataset where the number of features is much larger than the number of samples , fitting a model with all features is often computationally prohibitive or statistically invalid (e.g., for ordinary least squares, the solution is not unique). Forward selection is more feasible because it starts with zero features and adds them one by one, never needing to fit a model with more features than samples in its early stages.

Pearson correlation only measures the strength of a linear relationship. A feature can have a correlation of nearly zero with the target but still be highly predictive if its relationship is non-linear (e.g., a U-shaped relationship) or if it primarily contributes through interactions with other features. Tree-based models like Random Forest are excellent at capturing these complex, non-linear patterns, which would be reflected in a high importance score.

By construction, PCA finds a new set of axes (the principal components) for the data such that the components are orthogonal to each other. In a geometric sense, this means they are at right angles. In a statistical sense, this means their covariance is zero, making them linearly uncorrelated. This property is essential for reducing redundancy in the data representation.

The number of components that LDA can produce is limited by , where is the number of classes and is the number of features. In this case, and . Therefore, the maximum number of dimensions LDA can project the data onto is . LDA projects data onto a space that best separates the centers of the classes, and with C classes, there are at most C-1 such discriminatory dimensions.

The curse of dimensionality introduces numerous problems. It makes data sparse (B), requires exponentially more data to generalize well (C), and makes distance metrics less meaningful (D). It does not make models more interpretable. In fact, a model with hundreds or thousands of features is significantly less interpretable than a model with a few features. The other options are classic examples of the curse's effects.

is_long_session is directly derived from session_duration. This means the two features are highly correlated by definition. Including both in a linear model introduces multicollinearity, which can make the model's coefficient estimates unstable and difficult to interpret. While session_duration contains more information, the binary flag is a redundant representation of that same information.

The curse of dimensionality causes the volume of the feature space to increase so rapidly that the data points become very sparse. A key consequence is that the distance to the nearest neighbor and the farthest neighbor for a given point become almost equal. If is the distance to the farthest point and is the distance to the nearest, the ratio approaches 0. This makes it extremely difficult for algorithms like k-NN to distinguish between neighbors and non-neighbors based on distance, severely degrading their performance.

Adding a feature that is a perfect linear combination of others introduces perfect multicollinearity. This makes the new covariance matrix singular. A singular matrix has at least one eigenvalue equal to zero. The new feature doesn't add any new information (intrinsic dimensionality is unchanged), so the number of dimensions needed to capture the variance does not increase. The variance from the new feature will be absorbed into the existing principal components that already capture the variance from and , likely increasing the magnitude of the largest eigenvalues.

The number of discriminant components that LDA can produce is limited by the rank of the between-class scatter matrix . The rank of is at most , where is the number of classes. It is also limited by the number of features, . Therefore, the maximum number of components is . In this case, with 50 features and 3 classes, the maximum number of components is . The question asks for the formula using for features, which is often cited as a more precise bound for the rank of the within-class scatter matrix, but the dominant constraint here is from the number of classes, making the overall limit . Option D is the most precise and correct.

Forward Selection is a greedy algorithm. At each step, it evaluates adding each of the remaining features to the current set of selected features. If feature A has no predictive power on its own, it will not be selected in the first step. If B and C are selected, the algorithm will then test adding A to the set {B, C}. While in this case it might then add A, the question describes a more general limitation. The core problem is its 'myopic' or greedy nature. If A's contribution is only apparent when combined with B and C, and the algorithm happens to be at a stage with only {B} selected, adding A might still show no improvement. Thus, it can easily miss complex interactions and get stuck in a local optimum that does not include the true best subset.

Gini Importance (MDI) is calculated based on how much each feature contributes to reducing impurity within the trees during training. When two features are highly correlated, the model can use either one for a split and achieve a similar impurity reduction. This can lead to the importance being split between them, or more commonly, one feature gets chosen more frequently by chance, receiving a high importance score while the other gets a very low one. Permutation Importance, on the other hand, measures the drop in model performance when a feature's values are randomly shuffled. If monthly_income is shuffled but yearly_income is intact, the model can still use yearly_income to make good predictions, so the performance drop (and thus importance) for monthly_income might be small, potentially underestimating the true importance of the underlying concept of 'income'.

While F2 has a lower simple correlation with the target, this doesn't tell the whole story. It's possible that F1 is correlated with the target through the same mechanism as F2, but also through other mechanisms. F2 might contain unique information that is suppressed in the simple correlation analysis but would be valuable to a multivariate model like multiple regression. This is related to the concept of suppression effects. Automatically removing F2 based on the simple heuristic could lead to a loss of valuable, unique predictive information.

PCA is a variance-maximizing procedure. If features have different scales, the feature with the largest range and variance (like income in dollars) will overwhelmingly dominate the calculation of the first principal component. The algorithm will find that the direction of maximum variance is almost perfectly aligned with the axis of that single feature. Consequently, the first PC will essentially be a proxy for the 'income' feature, and the contributions from features with smaller variance (like 'age') will be negligible. This is why standardizing the data (to have a mean of 0 and a standard deviation of 1) is a critical preprocessing step for PCA.

Backward Elimination starts with the full model containing all p features. This allows it to see the combined effect and interactions of all features from the beginning. It then iteratively removes the least useful feature. Forward Selection starts with an empty model and adds features one by one, which can cause it to miss interactions that are only valuable when multiple specific features are present together. However, the starting point of Backward Elimination (fitting a model with all p features) can be extremely computationally expensive, or even impossible if p is very large or larger than the number of samples. The number of steps is at most p for both, but each step in Backward Elimination can be more costly, especially in the beginning.

After MinMaxScaler, all features are in the [0, 1] range, but their variances are not standardized. A feature whose values are bimodally distributed at the extremes (e.g., half the data is 0, half is 1) will have a high variance (around 0.25). A feature whose values are all clustered around 0.5 will have a very low variance. A uniform distribution on [0, 1] has a variance of 1/12 (approx 0.083). Therefore, applying a fixed variance threshold after min-max scaling can lead to the removal of features not because they are quasi-constant, but because their scaled distribution is concentrated in the middle of the range. This might incorrectly remove a genuinely useful feature.

LDA works by finding a projection that maximizes the ratio of between-class variance to within-class variance. The between-class variance is calculated based on the distance between the means of the classes. In the case of two concentric circles, the mean of the inner circle and the mean of the outer ring are both at the same point (the center). Therefore, the between-class scatter matrix () will be a zero matrix. LDA will be unable to find any direction that separates the class means, and the resulting projection will be useless for classification. This highlights a key limitation of LDA: it can only find linear separations.

A sharp elbow in the scree plot (or a rapid increase in cumulative variance that then plateaus) indicates that the first few principal components capture a large portion of the total variance, which happens when the original features are highly correlated. Conversely, a smooth, slow, linear-like increase in cumulative explained variance implies that each successive principal component adds only a small, roughly equal amount of variance. This occurs when the original features are largely uncorrelated (or orthogonal), and the total variance is spread out almost evenly across all dimensions. In such a case, dimensionality reduction with PCA is less effective, as many components are needed to retain most of the information.

A linear model is linear in its coefficients. By creating polynomial features (), we are transforming the feature space. The model is still of the form , which is a linear combination of these new features. Taylor's theorem states that any sufficiently smooth function (like ) can be approximated by a polynomial (its Taylor series). Therefore, by providing the model with polynomial features, we enable it to learn the weights () that form the coefficients of a Taylor polynomial that approximates . This is a fundamental concept in using linear models for non-linear problems.

This is a classic example of temporal data leakage. When you calculate the average for all Mondays in the dataset, the value you assign to a Monday in the training set is influenced by the sales of Mondays that occur after it, including those in the test set. At the time of prediction for a given day, you would not have access to future data. The correct, non-leaky approach is to use an expanding window or a rolling window. For any given day d, the avg_sales_for_day_of_week should be calculated using only data from days before d.

Feature extraction often involves a hierarchical process of creating increasingly abstract representations. For image data, a common pipeline is to first extract low-level features and then build higher-level features from them. Applying an edge detection filter like Sobel extracts basic information about gradients (edges). The Histogram of Oriented Gradients (HOG) then aggregates this gradient information into a more robust feature descriptor. This two-step process (low-level filter -> higher-level descriptor) is a classic computer vision feature extraction pipeline. The other options are either nonsensical (One-Hot Encoding pixels), redundant, or mix feature selection with extraction in an illogical order.

The 'concentration of norms' (or 'distance concentration') is a counter-intuitive geometric phenomenon in high-dimensional spaces. It states that as dimensionality increases, the Euclidean distance between any two randomly chosen points becomes very similar. The distribution of distances becomes very narrow (i.e., concentrated around the mean distance). This is the same issue described in the k-NN question from a more formal perspective. It undermines distance-based algorithms like k-NN, clustering (e.g., k-means), and kernel methods that rely on the notion of proximity, because the contrast between the distances to 'near' and 'far' points diminishes significantly.

Impurity-based measures are biased towards features with a high number of unique values (high cardinality). This is because such features offer more potential split points. With more potential splits to choose from, it's more likely that one of these splits will, by pure chance, reduce the impurity of the training data, even if the feature is not genuinely predictive. This can cause a high-cardinality feature (like a unique ID column, in the extreme case) to receive an inflated importance score. Permutation importance does not suffer from this particular bias.

A key symptom of multicollinearity in linear models is coefficient instability. When two or more features are highly correlated, the model has difficulty attributing the effect on the target variable to any single one of them. The coefficient estimates become very sensitive to small changes in the data or model specification. Removing one of the collinear features () forces the model to re-attribute the explanatory power that shared with its correlated counterparts. This causes the coefficients and statistical significance of those other features to change dramatically.

This question addresses the core trade-off between selection and extraction. Feature selection methods (like forward selection, backward elimination, LASSO) select a subset of the original features. This means the final model is built on understandable, real-world variables (e.g., 'age', 'income'). This is crucial for inference, where the goal is to understand the relationship between predictors and the outcome. Feature extraction methods like PCA create new, synthetic features (principal components) that are linear combinations of all original features. While these components are optimal for capturing variance, they are generally not interpretable in terms of the original problem domain, making PCA unsuitable when interpretability is the priority.

This question requires careful step-by-step application of a specific (though plausible) heuristic. First

This scenario describes a causal mediation model (A -> B -> Y). While a model using only feature A might still be predictively powerful (as A's influence on Y is partly carried through B), it obscures the true underlying mechanism. Feature B acts as a mediator. By removing it, you can no longer analyze the direct effect of A on Y versus the indirect effect that goes through B. If the goal is to understand how the system works (inference and interpretability), removing a mediating variable is a significant error, even if it doesn't harm predictive accuracy.

Stepwise methods like Forward Selection can be highly unstable. Small changes in the training data (like using a different subsample) can lead to large changes in the selected model (the set of chosen features). This instability is a major drawback of greedy selection procedures. It often occurs when there isn't a single, clear 'best' subset of predictors. Instead, there might be multiple, different subsets of features that achieve very similar performance, or when features are highly correlated, the algorithm might pick one feature in one run and its correlated counterpart in another run. This highlights the need for more robust selection methods like LASSO or using resampling techniques like bootstrapping to assess feature selection stability.

The curse of dimensionality causes the volume of the feature space to increase so rapidly that the data points become very sparse. A key consequence is that the distance to the nearest neighbor and the farthest neighbor for a given point become almost equal. If is the distance to the farthest point and is the distance to the nearest, the ratio approaches 0. This makes it extremely difficult for algorithms like k-NN to distinguish between neighbors and non-neighbors based on distance, severely degrading their performance.

Adding a feature that is a perfect linear combination of others introduces perfect multicollinearity. This makes the new covariance matrix singular. A singular matrix has at least one eigenvalue equal to zero. The new feature doesn't add any new information (intrinsic dimensionality is unchanged), so the number of dimensions needed to capture the variance does not increase. The variance from the new feature will be absorbed into the existing principal components that already capture the variance from and , likely increasing the magnitude of the largest eigenvalues.

The number of discriminant components that LDA can produce is limited by the rank of the between-class scatter matrix . The rank of is at most , where is the number of classes. It is also limited by the number of features, . Therefore, the maximum number of components is . In this case, with 50 features and 3 classes, the maximum number of components is .

Forward Selection is a greedy algorithm. At each step, it evaluates adding each of the remaining features to the current set of selected features. If feature A has no predictive power on its own, it will not be selected in the first step. Even if B and C are later selected, the greedy nature means it may have already missed the opportunity to see the interaction. Its 'myopic' view prevents it from seeing the benefit of combinations of features that are not individually strong, leading it to a suboptimal solution.

Gini Importance (MDI) is calculated during training. When two features are highly correlated, the model can use either for a split and achieve a similar impurity reduction. This can cause one feature to be chosen more frequently by chance, receiving a high importance score while the other gets a very low one. Permutation Importance, calculated post-training, measures the performance drop when a feature is shuffled. If monthly_income is shuffled but the correlated yearly_income is intact, the model can still use yearly_income to make good predictions. Thus, the performance drop (and thus importance) for monthly_income might be small, underestimating the true importance of the underlying 'income' concept.

While F2 has a lower simple correlation with the target, this doesn't tell the whole story. It's possible that F1 is correlated with the target through the same mechanism as F2, but also through other mechanisms. F2 might contain unique information that is suppressed in the simple correlation analysis but would be valuable to a multivariate model like multiple regression. This is related to the concept of suppression effects. Automatically removing F2 based on the simple heuristic could lead to a loss of valuable, unique predictive information.

PCA is a variance-maximizing procedure. If features have different scales, the feature with the largest range and variance (like income in dollars) will overwhelmingly dominate the calculation of the first principal component. The algorithm will find that the direction of maximum variance is almost perfectly aligned with the axis of that single feature. Consequently, the first PC will essentially be a proxy for the 'income' feature, and the contributions from features with smaller variance (like 'age') will be negligible. This is why standardizing the data (to have a mean of 0 and a standard deviation of 1) is a critical preprocessing step for PCA.

Backward Elimination starts with the full model containing all p features. This allows it to see the combined effect and interactions of all features from the beginning. It then iteratively removes the least useful feature. Forward Selection starts with an empty model and adds features one by one, which can cause it to miss interactions that are only valuable when multiple specific features are present together. However, the starting point of Backward Elimination (fitting a model with all p features) can be extremely computationally expensive, or even impossible if p is very large or larger than the number of samples.

After MinMaxScaler, all features are in the [0, 1] range, but their variances are not standardized. A feature whose values are bimodally distributed at the extremes (e.g., half the data is 0, half is 1) will have a high variance (around 0.25). A feature whose values are all clustered around 0.5 will have a very low variance. A uniform distribution on [0, 1] has a variance of 1/12 (approx 0.083). Therefore, applying a fixed variance threshold after min-max scaling can lead to the removal of features not because they are quasi-constant, but because their scaled distribution is concentrated in the middle of the range. This might incorrectly remove a genuinely useful feature.

LDA works by finding a projection that maximizes the ratio of between-class variance to within-class variance. The between-class variance is calculated based on the distance between the means of the classes. In the case of two concentric circles, the mean of the inner circle and the mean of the outer ring are both at the same point (the center). Therefore, the between-class scatter matrix () will be a zero matrix. LDA will be unable to find any direction that separates the class means, and the resulting projection will be useless for classification. This highlights a key limitation of LDA: it can only find linear separations.

A sharp elbow in the scree plot (or a rapid increase in cumulative variance that then plateaus) indicates that the first few principal components capture a large portion of the total variance, which happens when the original features are highly correlated. Conversely, a smooth, slow, linear-like increase in cumulative explained variance implies that each successive principal component adds only a small, roughly equal amount of variance. This occurs when the original features are largely uncorrelated (or orthogonal), and the total variance is spread out almost evenly across all dimensions. In such a case, dimensionality reduction with PCA is less effective, as many components are needed to retain most of the information.

A linear model is linear in its coefficients. By creating polynomial features (), we are transforming the feature space. The model is still of the form , which is a linear combination of these new features. Taylor's theorem states that any sufficiently smooth function (like ) can be approximated by a polynomial (its Taylor series). Therefore, by providing the model with polynomial features, we enable it to learn the weights () that form the coefficients of a Taylor polynomial that approximates . This is a fundamental concept in using linear models for non-linear problems.

This is a classic example of temporal data leakage. When you calculate the average for all Mondays in the dataset, the value you assign to a Monday in the training set is influenced by the sales of Mondays that occur after it, including those in the test set. At the time of prediction for a given day, you would not have access to future data. The correct, non-leaky approach is to use an expanding window or a rolling window. For any given day d, the avg_sales_for_day_of_week should be calculated using only data from days before d.

Feature extraction often involves a hierarchical process of creating increasingly abstract representations. For image data, a common pipeline is to first extract low-level features and then build higher-level features from them. Applying an edge detection filter like Sobel extracts basic information about gradients (edges). The Histogram of Oriented Gradients (HOG) then aggregates this gradient information into a more robust feature descriptor. This two-step process (low-level filter -> higher-level descriptor) is a classic computer vision feature extraction pipeline. The other options are either nonsensical (One-Hot Encoding pixels), redundant, or mix feature selection with extraction in an illogical order.

The 'concentration of norms' (or 'distance concentration') is a counter-intuitive geometric phenomenon in high-dimensional spaces. It states that as dimensionality increases, the Euclidean distance between any two randomly chosen points becomes very similar. The distribution of distances becomes very narrow (i.e., concentrated around the mean distance). This is the same issue described in the k-NN question from a more formal perspective. It undermines distance-based algorithms like k-NN, clustering (e.g., k-means), and kernel methods that rely on the notion of proximity, because the contrast between the distances to 'near' and 'far' points diminishes significantly.

Impurity-based measures are biased towards features with a high number of unique values (high cardinality). This is because such features offer more potential split points. With more potential splits to choose from, it's more likely that one of these splits will, by pure chance, reduce the impurity of the training data, even if the feature is not genuinely predictive. This can cause a high-cardinality feature (like a unique ID column, in the extreme case) to receive an inflated importance score. Permutation importance does not suffer from this particular bias.

A key symptom of multicollinearity in linear models is coefficient instability. When two or more features are highly correlated, the model has difficulty attributing the effect on the target variable to any single one of them. The coefficient estimates become very sensitive to small changes in the data or model specification. Removing one of the collinear features () forces the model to re-attribute the explanatory power that shared with its correlated counterparts. This causes the coefficients and statistical significance of those other features to change dramatically.

This question addresses the core trade-off between selection and extraction. Feature selection methods (like forward selection, backward elimination, LASSO) select a subset of the original features. This means the final model is built on understandable, real-world variables (e.g., 'age', 'income'). This is crucial for inference, where the goal is to understand the relationship between predictors and the outcome. Feature extraction methods like PCA create new, synthetic features (principal components) that are linear combinations of all original features. While these components are optimal for capturing variance, they are generally not interpretable in terms of the original problem domain, making PCA unsuitable when interpretability is the priority.

This scenario describes a causal mediation model (A -> B -> Y). While a model using only feature A might still be predictively powerful (as A's influence on Y is partly carried through B), it obscures the true underlying mechanism. Feature B acts as a mediator. By removing it, you can no longer analyze the direct effect of A on Y versus the indirect effect that goes through B. If the goal is to understand how the system works (inference and interpretability), removing a mediating variable is a significant error, even if it doesn't harm predictive accuracy.

Stepwise methods like Forward Selection can be highly unstable. Small changes in the training data (like using a different subsample) can lead to large changes in the selected model (the set of chosen features). This instability is a major drawback of greedy selection procedures. It often occurs when there isn't a single, clear 'best' subset of predictors. Instead, there might be multiple, different subsets of features that achieve very similar performance, or when features are highly correlated, the algorithm might pick one feature in one run and its correlated counterpart in another run. This highlights the need for more robust selection methods like LASSO or using resampling techniques like bootstrapping to assess feature selection stability.