Unit 4: Regression - Practice Quiz

Regression models are used for prediction tasks where the output variable is a continuous quantity, such as predicting a price, temperature, or height.

Predicting a house price involves forecasting a continuous numerical value, which is a regression task. The other options are classification tasks, as they involve assigning data to discrete, predefined categories.

High bias means the model is too simple and cannot capture the underlying patterns in the data, leading to underfitting. This results in high error on both training and test sets.

High variance is a characteristic of a model that is too complex and has learned the training data too well, including its random noise. This phenomenon is known as overfitting.

The term 'simple' in Simple Linear Regression refers to the use of a single independent variable to model a linear relationship with a single dependent variable.

is the slope, which represents the change in the dependent variable for a one-unit change in the independent variable .

Multiple Linear Regression extends Simple Linear Regression by allowing the use of multiple predictor variables () to predict a single outcome variable ().

This is the standard form for a multiple linear regression model, where is the intercept and and are the coefficients for the independent variables and , respectively.

Each coefficient in multiple regression represents the change in the dependent variable for a one-unit change in the corresponding independent variable, assuming all other variables are held constant.

A coefficient of zero means that a change in the predictor variable has no effect on the predicted outcome in the model, indicating the absence of a linear relationship as captured by the model.

Ridge (L2 regularization) and Lasso (L1 regularization) are the two most well-known techniques for adding a penalty term to the loss function of a linear regression model to prevent overfitting.

Regularization adds a penalty for model complexity (i.e., large coefficient values) to the cost function, which helps to reduce model variance and prevent overfitting the training data.

The regularization term penalizes the magnitude of the coefficients. A larger hyperparameter ( or ) means a stronger penalty, which forces the optimization process to find smaller coefficient values, thus reducing model complexity.

The L1 penalty used in Lasso Regression (sum of the absolute values of coefficients) has the property of shrinking less important feature coefficients all the way to zero, effectively removing them from the model.

By creating polynomial terms (like , , ), a linear model can fit more complex, non-linear patterns in the data without changing the underlying linear regression algorithm itself.

A polynomial expansion of degree 2 for a feature creates a new feature corresponding to raised to the power of 2, which is . The full set of features available to the model would be [1, , ].

A Decision Tree can be adapted for regression tasks (becoming a Decision Tree Regressor) by predicting a continuous value in its leaf nodes, typically the average of the target values of the training samples in that leaf.

Unlike a classification tree which predicts a class, a regression tree's leaf node predicts a single continuous value, which is usually the mean of the outcomes for the training data that ended up in that leaf.

Time-series data is a sequence of data points indexed in time order. This temporal dependence is a crucial aspect that must be considered during modeling.

A lag variable (or lag feature) is created by using a variable's value from a previous time step as a predictor for the current time step. A 'lag-1' feature is the value from the immediately preceding time step, e.g., using yesterday's sales to predict today's sales.

The core task is to predict a continuous numerical value (rainfall in mm), which is a regression problem. Logistic Regression is designed for classification tasks, where the goal is to predict a discrete outcome (e.g., 'Rain' or 'No Rain'). Its output is a probability that is mapped to a class, not a continuous quantity.

The large gap between a low training error and a high validation error is a classic sign of high variance, also known as overfitting. The model has learned the training data's noise instead of the underlying pattern. To remedy this, one can introduce regularization (like L2/Ridge) to penalize large coefficients or simplify the model (e.g., use fewer features or a less complex algorithm).

R-squared, or the coefficient of determination, measures the proportion of the variance in the dependent variable (house price) that is predictable from the independent variable(s) (square footage). An of 0.65 means that 65% of the variance in house prices is accounted for by the model.

Multicollinearity occurs when independent variables in a regression model are highly correlated. 'Age' and 'years_of_experience' are likely to be strongly correlated. When both are in the model, they explain the same variance in the target, making it difficult for the model to estimate their individual effects, resulting in high p-values and unstable coefficients. Removing one allows the other to capture that shared effect, making it appear significant.

In multiple regression, the coefficient of a variable represents the change in the predicted outcome for a one-unit change in that variable, assuming all other variables in the model are held constant. This 'ceteris paribus' condition is crucial for correct interpretation.

Lasso (Least Absolute Shrinkage and Selection Operator) Regression uses an L1 penalty term (), which has the property of shrinking some feature coefficients to exactly zero. This effectively removes the feature from the model, thus performing automatic feature selection. Ridge regression (L2) shrinks coefficients towards zero but never sets them exactly to zero.

The Ridge regression cost function is . As becomes very large, the penalty for having non-zero coefficients dominates the cost function. To minimize the cost, the model must shrink the coefficients () to be as close to zero as possible. In the limit, all coefficients become zero, and the model's prediction is simply the intercept, which is the mean of the target variable in a standardized setting.

The U-shaped relationship indicates a non-linear pattern that a simple linear model cannot capture (high bias). By performing a polynomial feature expansion and adding a quadratic term (), you allow the model to fit a curve (a parabola) to the data. This increases the model's complexity to better match the underlying relationship.

A single, deep decision tree is prone to overfitting, meaning it has high variance. A Random Forest builds multiple decision trees on different bootstrap samples of the data and considers only a random subset of features for each split. By averaging the predictions of these diverse (decorrelated) trees, it significantly reduces the overall variance of the model, leading to better generalization performance.

SARIMA models are an extension of ARIMA models specifically designed for time series data with a seasonal component. It includes parameters (P, D, Q, m) to model the seasonality in addition to the standard ARIMA parameters (p, d, q) for trend and autocorrelation. While a linear regression with time and dummy variables for months can capture some of this, SARIMA is built from the ground up to handle these complex time-series dynamics.

The extremely low training RMSE (2) for the degree-10 polynomial suggests it fits the training data almost perfectly, indicating low bias. However, its performance on the validation set is very poor (RMSE of 40), which is much worse than the simpler degree-2 model. This large gap between training and validation performance is a hallmark of high variance (overfitting).

The standard metric can only increase or stay the same when a new feature is added, regardless of its utility, because the model can simply assign it a coefficient of zero if it's useless. Adjusted , however, penalizes the addition of features that do not improve the model more than would be expected by chance. Therefore, adding a useless feature will cause the Adjusted to decrease.

The assumption of homoscedasticity states that the variance of the residuals should be constant across all levels of the independent variable(s). A funnel shape in the residual plot indicates that the variance of the errors is not constant; it increases (or decreases) as the predicted value changes. This violation is called heteroscedasticity.

When using dummy variables for categorical features, the coefficient of a specific level (e.g., 'Engineering') represents the average difference in the target variable between that level and the reference (or baseline) category ('Sales'), holding all other model variables constant.

This describes the regularization path of a Lasso model. As the penalty term increases, it puts more pressure on the model to have smaller coefficients. The L1 penalty used by Lasso has the unique effect of forcing some coefficients to become precisely zero once the penalty is strong enough, effectively removing them from the model.

When features are highly correlated, Lasso's behavior can be unstable; it often arbitrarily selects one feature from a group of correlated features and sets the coefficients of the others to zero. Ridge, on the other hand, tends to distribute the coefficient weight among the correlated features, shrinking them together. This can lead to a more stable and sometimes more interpretable model when you believe the correlated features are all relevant.

A high-degree polynomial creates a very flexible model with high complexity. While it might achieve a very low error on the training set by weaving through the data points, it is likely capturing noise rather than the true underlying function. This results in high variance (overfitting) and poor performance on unseen data.

A decision tree works by recursively partitioning the feature space. For a new data point, it follows the splitting rules from the root node down to a terminal (leaf) node. The prediction for any point that falls into that leaf node is simply the mean of the target variable ('y' values) of all the training data points that ended up in that same leaf during training.

An Autoregressive (AR) model is based on the idea that the current value of a time series is dependent on its own previous values. The 'p' in AR(p) specifies that the model uses the 'p' most recent time steps (lags) as predictors in a linear equation to forecast the next value.

Regression models predict continuous values, so their performance is measured by error metrics like RMSE, MAE, or MSE, which quantify the magnitude of the prediction errors. Classification models predict discrete classes, and their performance is often evaluated using metrics like Accuracy, Precision, Recall, F1-score, or AUC, which measure how well the model distinguishes between classes.

Ridge's penalty term () is minimized when the coefficients for correlated variables are close to each other (it 'prefers' to split the predictive power). Thus, it shrinks them together. Lasso's penalty () is indifferent to how the power is split and performs feature selection. Due to the geometry of the constraint, it will often find a solution where one coefficient is pushed to exactly zero, effectively selecting one variable from the correlated group.

A high-degree polynomial (degree-10) is a highly flexible model. On the training data, it can fit both the underlying linear trend and the random noise, resulting in low training error and thus low bias with respect to the training set. However, this fitting of noise means the model's parameters will vary wildly with different samples of training data, leading to very high variance. This high variance results in poor performance on unseen test data. The simple degree-1 model would have low variance but slightly higher bias (as it cannot fit the noise perfectly).

GBR is a sequential, boosting ensemble method that aims to reduce bias by fitting new models to the residuals of previous models. Overfitting in GBR occurs when it starts fitting the noise in the residuals, leading to extremely low bias on the training data but high variance. RFR is a parallel, bagging ensemble method that averages many deep, high-variance, low-bias trees. The averaging reduces variance. Overfitting in RFR occurs when the individual trees are too complex and the averaging process is insufficient to cancel out the variance, often because the trees are too correlated.

The coefficient represents the interaction effect. The marginal effect of Size on Price is the partial derivative of Price with respect to Size, which is . This shows the effect of size is not constant but depends on age. The interpretation of is how this marginal effect changes for a one-unit change in Age. Specifically, for each one-unit increase in Age, the slope of Price with respect to Size changes by .

The Durbin-Watson statistic tests for first-order autocorrelation in residuals. A value near 2 indicates no autocorrelation. Values approaching 0 indicate strong positive autocorrelation. A value of 0.35 is very close to 0, indicating a severe violation of the OLS assumption of independent errors. A major consequence of positive autocorrelation is that the standard errors of the regression coefficients will be underestimated, leading to overly optimistic (inflated) t-statistics and p-values. This makes coefficients appear more statistically significant than they actually are.

The effective degrees of freedom of a model measure its complexity. For an unregularized linear model with predictors, the effective degrees of freedom is . As the regularization penalty increases, the coefficients are constrained and shrunk towards zero. This reduces the model's flexibility to fit the data, thus lowering its complexity. As , all coefficients are forced to zero (for a model without an intercept), and the model becomes a null model with 0 effective degrees of freedom. Therefore, the effective degrees of freedom decrease from towards 0 as increases.

This is a case of perfect multicollinearity, where one predictor is a perfect linear combination of another. The OLS solution for the coefficients is given by . When perfect multicollinearity exists, the columns of the design matrix are linearly dependent. This causes the matrix to be singular (or non-invertible), meaning its determinant is zero. As a result, its inverse is not defined, and the OLS procedure fails to find a unique set of coefficients. While some software packages implement a practical workaround (like option C), the fundamental mathematical result is the non-invertibility of .

Ridge regression minimizes the objective function . As the regularization parameter becomes extremely large, the penalty for non-zero coefficients dominates the RSS term. To minimize this objective, the model must force all slope coefficients () to approach zero. The intercept term, , is typically not regularized. The model that minimizes the RSS with all slope coefficients being zero is a model that predicts the mean of the target variable for all inputs. Therefore, the function will approximate a constant function, where .

The Gauss-Markov theorem states that under the standard OLS assumptions (linearity, exogeneity, homoscedasticity, and no perfect multicollinearity), the OLS estimator is BLUE. Each word has a specific meaning: "Linear" means it's a linear function of the observed values. "Unbiased" means its expected value is the true population parameter (). "Best" specifically means that it has the lowest sampling variance compared to any other linear unbiased estimator. This implies that OLS estimates are the most precise or reliable in this class of estimators.

Standard regression predicts a continuous value (e.g., price, temperature). Standard classification predicts a discrete, categorical label (e.g., cat, dog, bird). A Poisson regression model uses a linear model to predict the logarithm of a continuous, positive real number, the rate parameter (the expected number of events). This prediction of a continuous parameter is a regression-like task. However, this parameter then defines a probability distribution over a discrete, countably infinite set of outcomes (the integers 0, 1, 2, ...). This makes the target variable discrete, which is a characteristic of classification. Therefore, it sits in a gray area, using regression techniques to model a discrete (count) outcome.

The eta (learning rate) scales the contribution of each new tree added to the ensemble. A small eta (e.g., 0.01) means each tree makes a very small correction to the overall model. This prevents the model from making drastic changes based on any single tree, which helps avoid overfitting. However, since each step is small, many more steps (n_estimators) are required for the model to 'travel' to a good minimum in the loss function. This combination allows for a more robust and fine-tuned model that generalizes better by taking smaller, more careful steps.

Elastic Net combines the penalty of Lasso and the penalty of Ridge. The part enables sparse solutions (feature selection). The part encourages correlated features to be selected together, giving them similar coefficient values. This is known as the "grouping effect." Lasso, by itself, tends to arbitrarily select one feature from a correlated group and zero out the others. Ridge will keep all correlated features but won't perform feature selection. PCR handles correlation but by transforming features into uncorrelated components, losing original feature interpretability. Elastic Net is specifically designed for sparse selection within groups of correlated features.

Standard metrics like MSE penalize over- and under-predictions equally. The business problem describes an asymmetric cost. High-cost events (large defaults) are often in the tails of the distribution. A high-bias, low-variance model (e.g., a simple linear model) might be stable but could systematically under-predict these extreme values. A more complex, low-bias, high-variance model (e.g., a complex GBT or a well-tuned neural network) is more flexible and has a better chance of capturing these rare but critical high-default events. The priority is to avoid catastrophic under-prediction, which means accepting a more complex model that can produce large predictions, even at the risk of higher variance.

This is a log-log model, and the coefficient represents the elasticity of with respect to . Mathematically, . This ratio of percentage changes is interpreted as: a 1% change in is associated with a % change in . So, a 1% increase in advertising spend is associated with an expected 0.8% increase in sales. While option D is mathematically literal, it is not the standard, practical interpretation used to convey the business impact.

The Lasso path algorithm (like LARS) effectively builds the model sequentially. As you relax the penalty (decrease ), the algorithm allows coefficients to become non-zero one by one. The feature whose coefficient 'peels off' from zero first is the one that has the strongest correlation with the current residuals. This sequence of entry into the model provides a powerful heuristic for ranking feature importance. Features that can withstand stronger penalties (higher ) before being zeroed out are generally considered more important by the model.

Leverage is a measure determined solely by the predictor variables ( values). A point has high leverage if its values are far from the center of the other values. Influence is a measure of how much the model's parameters (e.g., coefficients) change when a point is removed. Influence is a function of both leverage and the size of the point's residual. A point with high leverage can be non-influential if it lies close to the regression line determined by the other points (i.e., it has a small residual). However, a high-leverage point with a large residual will almost always be highly influential.

Consider income () on a scale of and age () on a scale of . The polynomial term will be on the order of , while will be on the order of . When these terms are fed into a linear model, the huge disparity in scale can lead to numerical precision issues. Furthermore, for regularized models like Ridge or Lasso, the single penalty term is applied to all coefficients. The penalty will be dominated by the coefficients of the large-scale features, effectively ignoring the small-scale features. Scaling first (e.g., using StandardScaler) ensures all original and generated polynomial features are on a comparable scale, leading to stable training and fair regularization.

A key assumption for many time-series models, including AR(p), is that the underlying series is stationary (i.e., its statistical properties like mean and variance do not change over time). A series with a trend and seasonality is non-stationary. Fitting a model directly to such data can lead to a spurious regression. The model might find a very high R-squared and seemingly significant coefficients simply because the target () and the lagged features () are all increasing together due to the common trend. This relationship is not real and will break down for out-of-sample forecasting. The standard practice is to first difference the series (and/or take seasonal differences) to make it stationary before fitting the model.

Tree-based models, including Random Forests, cannot extrapolate beyond the range of the training data. A prediction is made by averaging the values in the terminal leaves where a new data point falls. For a value of X=200, which is outside the training range of [0, 100], the data point will traverse each tree down a path. At any split on X, since 200 is greater than any split point based on X in the training data, it will always follow the same branch as a data point with the maximum X value from training. It will therefore end up in the same terminal leaves as the training points with the largest X values. The final prediction will be the average of the target values in those leaves, effectively capping the prediction at the level seen at the edge of the training data.

The solution to a regularized regression problem is the point where the elliptical contours of the RSS (centered at the OLS solution) first touch the boundary of the constraint region defined by the penalty. For Ridge regression ( penalty), the constraint is a circle (in 2D), a smooth shape. For Lasso ( penalty), the constraint is a rhombus (a diamond shape in 2D). This shape has sharp corners that lie exactly on the axes (e.g., at or ). It is geometrically probable that the expanding RSS ellipse will hit one of these sharp corners before it touches any of the flat sides. A point on a corner means one of the coefficients is exactly zero, thus inducing sparsity.