Unit 1 - Practice Quiz

Nominal data represents categories that do not have an intrinsic order or ranking. Colors like Red, Blue, and Green are distinct categories with no inherent order, unlike satisfaction ratings or sizes which are examples of Ordinal data.

Discrete data can only take specific, countable values (like integers). You can have 30 or 31 students, but not 30.5 students. Continuous data, like height or weight, can take any value within a given range.

Data pre-processing is the crucial first step where raw data is cleaned, transformed, and prepared to make it suitable for training and building machine learning models. It improves the quality of the data, and thereby, the model's performance.

Handling missing values, encoding categorical variables, and feature scaling are all common and essential data pre-processing techniques. Model evaluation and deployment are later stages in the machine learning lifecycle, performed after the model has been trained.

Data leakage occurs when the training data contains information about the target that would not be available at the time of prediction. This leads to an overly optimistic model performance during training but poor performance in production.

The data should be split into training and testing sets first. Pre-processing steps like scaling or imputation should then be fitted on the training data only and subsequently applied to both the training and testing data to avoid information from the test set 'leaking' into the training process.

Imputation is the process of replacing missing data with substituted values. Common methods include using the mean, median, or mode of the respective column.

The mean is highly sensitive to outliers, so using it for imputation can skew the data. The median is a more robust measure of central tendency and is less affected by extreme values, making it a better choice in this scenario.

The simplest approach is to remove the entire row (or column) that contains missing values. This is known as listwise deletion. However, this can lead to a significant loss of data if many rows have missing values.

An outlier is an observation that lies an abnormal distance from other values in a random sample from a population. It is an extreme value that deviates from other observations.

Outliers can have a disproportionate effect on statistical measures like the mean and standard deviation, and they can negatively impact the training process of many machine learning algorithms, leading to a less accurate model.

One-Hot Encoding is a process that creates a new binary (0 or 1) column for each unique category. This is done to represent categorical data in a numerical format that machine learning algorithms can understand, without implying any order between categories.

Label Encoding assigns a unique integer to each category (e.g., Small=1, Medium=2, Large=3). This implies an order, which is perfect for ordinal data. Using it for nominal data can mislead the model into thinking there is a ranking where none exists.

Feature scaling is a method used to standardize the range of independent variables or features of data. Since the range of values of raw data varies widely, some machine learning algorithms might not work correctly without it, as features with larger ranges can dominate the learning process.

Min-Max Scaling, also known as normalization, rescales the features to a fixed range, which is typically 0 to 1. The formula is: .

Standardization rescales data so that it has a mean () of 0 and a standard deviation () of 1. The formula for a data point is .

Class imbalance occurs when one or more classes in a classification dataset have a very low number of samples (the minority class) compared to another class (the majority class). For example, a fraud detection dataset might have 99% non-fraudulent and 1% fraudulent transactions.

Oversampling techniques aim to balance the dataset by increasing the size of the minority class. SMOTE (Synthetic Minority Over-sampling Technique) is a popular method that creates new synthetic samples rather than just duplicating existing ones.

The Interquartile Range (IQR) is the range between the first quartile (25th percentile) and the third quartile (75th percentile). A common rule of thumb is to identify data points as outliers if they fall below or above .

If a categorical variable has a large number of unique categories (high cardinality), One-Hot Encoding will create an equally large number of new features. This can significantly increase the dimensionality of the dataset, which can make model training more computationally expensive and sometimes lead to poorer performance.

For a skewed distribution, the mean is heavily influenced by outliers and extreme values, making it a poor representation of the central tendency. The median is robust to outliers and skewness, making it a much better choice for imputing missing values in such cases. Mode imputation is for categorical data, and dropping rows would lead to loss of valuable information.

This is a classic example of data leakage. The scaler learns the parameters (mean and standard deviation) from the entire dataset, including the test set. This information then 'leaks' into the training process, leading to an overly optimistic performance estimate on the test set, as the model has already 'seen' some of its statistical properties.

Linear models interpret numerical inputs as having magnitude and order. Label Encoding assigns arbitrary integers (e.g., USA=0, UK=1, Canada=2), which implies that Canada > UK > USA. The model would incorrectly learn a relationship based on this artificial order. One-Hot Encoding avoids this by creating binary columns for each category, representing them as distinct entities without any implied ranking.

RobustScaler is specifically designed to handle outliers. It scales features using statistics that are robust to outliers, namely the median and the interquartile range (IQR). StandardScaler (using mean and std dev) and MinMaxScaler (using min and max) are both sensitive to outliers, which can skew the transformed data.

The test set must remain a true, unseen representation of the real-world data distribution. If SMOTE is applied before splitting, synthetic data points generated from the training set could end up in the test set, leading to data leakage. If applied to the test set, you are evaluating the model on an artificially altered distribution, not the real one. Therefore, resampling should only ever be done on the training data.

The Z-score method relies on the mean and standard deviation, which are sensitive to outliers and assume a roughly normal distribution. The IQR method is based on percentiles (the 25th and 75th), making it non-parametric and robust to the underlying distribution shape and the presence of outliers.

This is ordinal data because the values are categories that can be logically ranked (Excellent > Good > Average > Poor). However, the interval between the categories is not guaranteed to be equal or quantifiable (the difference between 'Good' and 'Excellent' isn't necessarily the same as between 'Poor' and 'Average').

Target leakage occurs when a feature's value would not be known at the time of prediction in a real-world scenario. Since the treatment is given after the diagnosis (the target variable), it leaks information from the future into the model, leading to an artificially high performance that will not generalize.

For high-cardinality features, techniques that reduce dimensionality are preferred. Binary Encoding is a good compromise. It first converts categories to integers (like Label Encoding) and then converts those integers to binary code. Each binary digit then becomes a feature. This creates far fewer features than One-Hot Encoding (log₂(n) features instead of n). Label Encoding is unsuitable for nominal data in many models, and dropping the feature might discard valuable information.

Random Forest is a tree-based algorithm. At each node, a decision tree splits a feature based on a threshold value. The choice of the split point is not affected by the scale of the feature; it only depends on the ordering of the values. Therefore, algorithms like Random Forest and other decision tree variants are not sensitive to the scale of the input features. In contrast, distance-based (KNN, SVM) and variance-based (PCA) algorithms are highly sensitive to feature scaling.

Missing At Random (MAR) is the condition where the missingness of a variable can be explained by other observed variables in the dataset. In this case, the probability of 'Salary' being missing is related to the 'Level of Education' variable. This is distinct from MCAR (purely random) and MNAR (where missingness depends on the unobserved value itself).

SMOTE's key innovation over simple random oversampling is that it generates new, synthetic data points. It selects a minority class instance, finds its k-nearest minority class neighbors, and creates a synthetic point along the line segment joining the instance and one of its selected neighbors. This helps to create more general decision boundaries.

A log transformation is a variance-stabilizing transformation. For right-skewed data with large outliers, it compresses the range of the larger values more than the smaller values. This reduces the skewness, makes the distribution more symmetric (often closer to normal), and lessens the leverage that extreme outliers have on models like linear regression, without completely discarding their information.

The cardinal rule of pre-processing is to prevent data leakage from the test set to the training set. The data must be split first. Then, any data-dependent transformations (like fitting a scaler or an imputer) must be learned only from the training data. The same learned transformation is then applied to the test data to ensure consistency without leaking information.

Target encoding is prone to overfitting because categories with few samples will have their encoded value determined by just a handful of target values, making them noisy and unreliable. Smoothing addresses this by calculating a weighted average between the category's mean and the overall global mean. For rare categories, the encoded value will be closer to the global mean, making it more robust.

This is the fundamental difference. StandardScaler centers the data around the mean with a unit standard deviation (Z-score normalization). MinMaxScaler rescales the data to a specific range by shifting and scaling, using the minimum and maximum values in the data. The choice between them depends on the algorithm and the data's properties (e.g., presence of outliers).

Time-series data has an inherent temporal order. Methods like LOCF (using the last known value) or linear/spline interpolation respect this order by using adjacent time steps to estimate the missing value. Mean/median imputation ignores the temporal dependency, and KNN imputation can be computationally expensive and may not respect the time flow.

This is known as the accuracy paradox. A naive model that always predicts the majority class ('not fraud') would achieve 99.9% accuracy but would be completely useless, as it would never identify any fraudulent transactions. In such cases, metrics like Precision, Recall, F1-Score, or the AUC-ROC are far more informative because they evaluate the model's ability to correctly identify the rare, minority class.

'Postal code' is a classic example of a nominal variable that happens to be represented by numbers. The numbers are labels for geographic areas. There is no inherent order or magnitude in the numbers themselves; the difference between two codes is meaningless. Treating it as a continuous variable would cause a model to learn incorrect relationships based on this artificial magnitude.

Decision trees make splits on features to partition the data. An outlier might be partitioned into its own leaf, but its large value does not affect the splits in other parts of the tree. This makes them robust to outliers. In contrast, Linear Regression, SVMs, and K-Means Clustering are all sensitive to outliers because they are based on minimizing distances or errors, and outliers can exert significant leverage on the final model or cluster centers.

Data leakage occurs when information from outside the training dataset is used to create the model. In a cross-validation setting, each fold serves as a hold-out test set once. By calculating the mean and standard deviation for each profession using the entire dataset, the feature values for samples in the training folds are influenced by the values of samples that will be in the validation fold. The correct approach is to compute these statistics only from the training data within each fold of the cross-validation loop and then apply that transformation to the validation data.

Target Encoding directly uses the target variable to create the encoding, which carries a high risk of overfitting. For a category with only one sample, the encoded value would be either 0 or 1, causing the model to perfectly 'memorize' the training label. Applying a smoothing factor (e.g., blending the category's mean with the global mean of the target) is crucial. This technique, often implemented as part of a careful cross-validation strategy, ensures that the encoding for rare categories is regularized towards the global average, mitigating the risk of the model relying too heavily on noisy, high-variance estimates from small groups.

MICE works by building a series of regression models to predict missing values based on the other features. Its core assumption is that the 'missingness' can be explained by the observed values in other columns (MAR). In an MNAR scenario, the reason for missingness depends on the unobserved value itself (e.g., very high values are missing because they are high). MICE, without being explicitly modeled to handle the MNAR mechanism (e.g., through a selection model), will impute values based on the distribution of the observed data, leading to biased estimates that don't reflect the true underlying distribution of the complete data.

SMOTE creates synthetic samples by interpolating between a minority sample and its k-nearest minority neighbors. If a minority sample is noisy or located deep within the majority class region, its nearest minority neighbors might be far away, across a dense region of majority samples. Interpolating between them can create synthetic data points that are ambiguous and fall in areas where the class boundary should be sharp. This process is called 'overgeneralization' and can introduce more noise, making it more difficult for a classifier to learn a clear separation.

SVMs with RBF kernels are distance-based algorithms and are highly sensitive to feature scaling. The presence of significant outliers makes StandardScaler (which uses mean and standard deviation) and MinMaxScaler (which uses min and max) poor choices. Outliers will heavily skew the mean/std dev and the min/max values, causing the majority of the data points to be compressed into a very small range. RobustScaler is the best choice because it scales data based on the median and the interquartile range (IQR), which are statistics robust to outliers. This ensures that the scaling reflects the bulk of the data's distribution, not the extremities.

Winsorization replaces extreme values with the value at a certain percentile (e.g., the 98th percentile). This means all values above the 98th percentile are set equal to the value at the 98th percentile. Capping at a fixed threshold can be arbitrary and might create an artificial cluster of data points at that threshold if it's not chosen carefully. Winsorization, by using a data-driven percentile, is less arbitrary. More importantly, it clips the tail without completely removing it, which can be less disruptive to the overall variance and relationships with other variables compared to creating a large, artificial spike at a single capped value.

The original approach uses future data (days t+1, t+2, t+3) to create a feature for predicting the target at time t. In a real-world prediction scenario, this future information would be unavailable. This is a severe form of data leakage. The correct, non-leaky approach is to use a trailing (or lagged) window, where all feature calculations for a given time point t rely only on information that was available at or before t-1. Using data from t-7 to t-1 ensures the feature is based purely on past information.

In a linear regression model that includes an intercept term, using OHE (k dummy variables) creates perfect multicollinearity. This is because the sum of the OHE columns for any observation is always 1, which is perfectly correlated with the intercept column (a column of all 1s). This condition, known as the dummy variable trap, means there is no unique solution for the model's coefficients. Dummy Coding (k-1 variables) breaks this perfect correlation by dropping one category (the reference category), allowing the model to estimate stable and interpretable coefficients for the remaining categories relative to the reference.

The key difference between SMOTE and ADASYN is how they select minority points for synthesis. SMOTE generates an equal number of synthetic samples for each minority instance. ADASYN, on the other hand, is adaptive. It calculates a ratio based on the number of majority class samples in the neighborhood of each minority sample. Minority samples in 'harder-to-learn' regions (i.e., closer to the decision boundary, with more majority neighbors) are assigned a higher weight, and consequently, more synthetic samples are generated around them. This adaptively shifts the decision boundary to focus on the difficult-to-classify examples.

QuantileTransformer is a powerful non-linear transformation that maps the cumulative distribution function (CDF) of a feature to a target distribution (either uniform or normal). By mapping data points to their quantiles, it effectively forces any input distribution into the desired one. This is particularly useful for distance-based algorithms (like k-NN or SVMs) or models that make assumptions about feature distributions, as it can spread out the most frequent values and reduce the influence of both inliers and outliers.

LOCF propagates a single value across multiple subsequent time steps. This creates flat, constant segments in the time series wherever data was missing. This process artificially reduces the true variability of the series because the imputed values do not fluctuate as they naturally would. Furthermore, it creates a false autocorrelation structure, as the imputed values are perfectly correlated with the last observed value. This can mislead time-series models and cause them to produce overly optimistic performance metrics and narrower, incorrect confidence intervals for forecasts.

Distance-based outlier detection methods like DBSCAN, LOF, or k-NN rely on calculating distances (e.g., Euclidean) between points. In high-dimensional spaces, the concept of 'neighborhood' or 'density' becomes less meaningful because the distance between any two points tends to become very similar (curse of dimensionality). Isolation Forest avoids this entirely. It works by randomly partitioning the feature space. Outliers, being 'few and different,' are easier to isolate and thus tend to be found closer to the root of the trees (shorter path lengths), regardless of the dimensionality. This makes its performance more robust in high-dimensional settings.

This feature is an example of ordinal data: there is a clear order, but the intervals between the values are not guaranteed to be equal. Treating it as a continuous numerical variable and applying StandardScaler () is flawed because the calculations of mean () and standard deviation () assume the data is on an interval or ratio scale where differences and averages are meaningful. The assumption that the difference in satisfaction between 'Very Unsatisfied' (1) and 'Unsatisfied' (2) is identical to the difference between 'Satisfied' (4) and 'Very Satisfied' (5) is often invalid and can introduce incorrect information into the model.

The correct order is critical. 1) Imputation: This should be done first on the original features. If you encode first, a single missing value in a categorical column turns into multiple missing values across all the new binary columns, making imputation much harder. 2) One-Hot Encoding: This converts the now-complete categorical feature into numerical binary columns. 3) Scaling: This should be last. If you scale before encoding, the scaler learns from the distribution of the original numerical features but has no knowledge of the new [0, 1] columns. Scaling the one-hot encoded columns after they are created ensures all features are on a comparable scale for the logistic regression model. Applying scaling before imputation is also wrong because the mean used for imputation would be 0 (from the scaled training data), which might not be a sensible imputation value.

Hash Encoding (or the 'hashing trick') shines in specific, constrained environments. Its primary advantages are: 1) It controls the output dimensionality to a fixed size, regardless of the number of unique categories, making it extremely memory-efficient for high-cardinality features. 2) It can handle new, unseen categories 'on the fly' without needing to refit the encoder, which is ideal for online/streaming systems. OHE would require adding a new column for every new category, which is infeasible. The main drawback of hashing is hash collisions (different categories mapping to the same hash), but in many large-scale applications, this information loss is an acceptable trade-off for the memory and processing gains.

Adjusting class weights does not change the data itself. Instead, it modifies the loss function. When calculating the loss during training, the errors made on samples from the minority class are multiplied by a larger weight, while errors on majority class samples get a smaller weight. For a logistic regression model, this means that the model is more heavily penalized for misclassifying a rare positive (fraud) case than for misclassifying a common negative (non-fraud) case. This effectively pushes the decision boundary away from the minority class samples to classify them more accurately, at the potential cost of misclassifying more majority class samples.

KNNImputer works by finding the k-nearest neighbors to a sample with missing values and then imputing the missing value based on the average (or mode) of the values from those neighbors. The 'nearest' is determined by a distance metric (e.g., Euclidean distance). If features are on vastly different scales (e.g., 'age' from 20-60 and 'income' from 30,000-200,000), the 'income' feature will completely dominate the distance calculation. The algorithm will find neighbors that are close in terms of income, ignoring all other features. This leads to poor, biased imputation because the concept of 'similarity' is skewed. Therefore, scaling features before using KNNImputer is a critical pre-processing step.

Sparsity is a valuable property, especially for memory and computation. StandardScaler calculates the mean of each feature and subtracts it from every entry. Since the mean of a sparse feature is typically a small non-zero number, this operation will turn almost all zero values into non-zero values, destroying the matrix's sparsity and dramatically increasing memory requirements. MaxAbsScaler and Normalizer are better choices as they scale by dividing by the maximum absolute value or by the norm, respectively. These operations preserve the zero entries, thus maintaining the data's sparsity.

This combination is particularly powerful. A Tomek Link is a pair of instances of opposite classes who are each other's nearest neighbors. These pairs are often considered noise or borderline cases. By removing the majority class sample from each Tomek Link, you 'clean' the space around the minority class, making the class boundary clearer. After this cleaning step, applying SMOTE is more effective because it is less likely to generate synthetic samples in noisy, overlapping regions. This two-step process (clean first, then generate) is a well-regarded strategy for improving classifier performance on imbalanced and noisy data.

This is a subtle but critical form of data leakage in time-series-like problems. When you randomly shuffle the data, your training set for a given fold will contain data points from the future relative to the data points in your validation set. The model inadvertently learns patterns or distributions from the future that help it predict the past/present, leading to an artificially inflated CV score. The proper way to validate time-dependent data is with a time-based split, such as TimeSeriesSplit or a simple train/validation split on the time axis (e.g., train on 2020, validate on 2021), which mimics the real-world scenario of predicting the future based only on the past.