Unit 1: Introduction and Data Preparation - Subjective Questions

Predictive analytics is the process of using historical and current data, statistical techniques, data mining, and machine learning algorithms to predict future events or unknown outcomes.

Major objectives include:

• Forecasting: Estimating future values such as sales, demand, or prices.
• Classification: Assigning observations to predefined categories, such as fraudulent or legitimate transactions.
• Risk assessment: Estimating the probability of undesirable outcomes such as loan default.
• Pattern discovery: Identifying relationships and trends hidden in data.
• Decision support: Helping organizations make informed and proactive decisions.

A predictive model generally represents the target variable as a function of input variables:

where is the learned relationship and represents unexplained variation.

A predictive analytics project usually consists of the following stages:

1. Problem definition: Clearly specify the business problem, prediction target, constraints, and success criteria.
2. Data collection: Gather relevant data from databases, files, sensors, surveys, or external sources.
3. Data understanding: Examine data types, distributions, relationships, quality, and possible biases.
4. Data preprocessing: Handle missing values, duplicates, outliers, inconsistent formats, and categorical variables.
5. Feature engineering: Create, transform, or select variables that improve predictive performance.
6. Model selection and training: Choose suitable algorithms and train them using historical data.
7. Model evaluation: Measure performance on unseen data using appropriate metrics.
8. Deployment: Integrate the selected model into an operational system.
9. Monitoring and maintenance: Track accuracy, data drift, and model performance over time.

These stages are iterative because evaluation results may require revisiting preprocessing, feature engineering, or model selection.

Machine learning is a branch of artificial intelligence in which computer systems learn patterns from data and use those patterns to make predictions or decisions without being explicitly programmed for every situation.

Principal types of machine learning:

• Supervised learning: Learns from labeled examples. For instance, a model may learn from historical loan records labeled as default or non-default.
• Unsupervised learning: Works with unlabeled data to discover hidden structures. Customer segmentation using clustering is an example.
• Semi-supervised learning: Uses a small amount of labeled data together with a large amount of unlabeled data. It is useful when labeling is expensive.
• Reinforcement learning: An agent learns by interacting with an environment and receiving rewards or penalties. Examples include robot navigation and game-playing systems.

The appropriate type depends on the availability of labels, the nature of the problem, and the desired output.

Supervised and unsupervised learning differ in the following ways:

In supervised learning, the model learns a mapping . In unsupervised learning, only is available, so the algorithm attempts to identify the underlying structure of the data.

Classification and regression are the two major supervised learning tasks.

Classification:

• Predicts a discrete or categorical target variable.
• The result belongs to one of a fixed set of classes.
• Examples include spam detection, disease diagnosis, and customer churn prediction.
• Common algorithms include logistic regression, decision trees, support vector machines, and naive Bayes.

Regression:

• Predicts a continuous numerical target variable.
• Examples include predicting house prices, sales revenue, temperature, or delivery time.
• Common algorithms include linear regression, regression trees, and support vector regression.

For simple linear regression, the predicted value is:

Thus, classification answers questions such as which category?, while regression answers questions such as how much? or what value?

Clustering is an unsupervised learning technique that divides observations into groups called clusters. Objects within the same cluster should be more similar to one another than to objects in other clusters.

Role and characteristics:

• It operates without predefined class labels.
• It reveals natural groupings and hidden structures in data.
• Similarity is commonly measured using a distance measure such as Euclidean distance:

• Common clustering methods include K-means, hierarchical clustering, and DBSCAN.

Applications include:

• Customer segmentation
• Grouping similar documents
• Image segmentation
• Detection of unusual observations
• Organization of biological data

Clustering results depend on feature selection, scaling, distance measures, and algorithm parameters. Therefore, preprocessing is especially important before clustering.

Data preprocessing is the process of converting raw data into a clean, consistent, and model-ready form before applying statistical or machine learning techniques.

It is essential because real-world data may contain:

• Missing values
• Duplicate records
• Incorrect or inconsistent entries
• Noise and outliers
• Variables measured on different scales
• Categorical values that algorithms cannot directly process
• Irrelevant or redundant features

Benefits of preprocessing:

• Improves model accuracy and reliability.
• Reduces misleading patterns caused by poor-quality data.
• Makes model training faster and more stable.
• Ensures that features are represented in a suitable form.
• Reduces the risk of biased or invalid conclusions.

Since a model learns directly from the supplied data, low-quality input generally produces low-quality predictions. Data preprocessing is therefore a critical stage rather than an optional preliminary task.

Missing values can be handled using several methods, depending on their frequency, cause, and the type of variable.

Common methods are:

• Record deletion: Remove rows containing missing values when the number of affected records is small and the missingness is random.
• Column deletion: Remove a feature when most of its values are missing and it is not essential.
• Mean imputation: Replace a missing numerical value with the feature mean. It is simple but sensitive to outliers.
• Median imputation: Replace it with the median. This is more robust for skewed data.
• Mode imputation: Use the most frequent category for a categorical variable.
• Group-based imputation: Calculate the replacement within a relevant group, such as the median income by occupation.
• Forward or backward filling: Use nearby observations, especially in ordered or time-series data.
• Model-based imputation: Predict missing values using regression, K-nearest neighbors, or multiple imputation.
• Missing-value indicator: Add a binary feature showing whether the original value was missing.

The imputation method must be learned from the training data and then applied to validation and test data to prevent data leakage.

An outlier is an observation that differs substantially from most other observations in a dataset. It may result from a measurement error, data-entry error, rare event, or valid extreme behavior.

Detection methods:

• Box plot and interquartile range: A value may be treated as an outlier if it lies below or above , where .
• Z-score: Values with a large absolute standardized score, often , may be considered unusual.
• Scatter plots: Useful for visually detecting extreme or isolated observations.
• Domain rules: Business or scientific knowledge can identify impossible values.
• Algorithmic methods: Isolation Forest, local outlier factor, and clustering can detect complex anomalies.

Treatment methods:

• Correct data-entry or measurement errors.
• Remove invalid observations.
• Cap extreme values using lower and upper limits.
• Apply transformations such as the logarithm.
• Use robust statistics or models.
• Retain valid outliers when they represent important rare events.

Outliers should not be removed automatically because they may contain valuable predictive information.

Normalization and standardization transform numerical features to comparable scales, but they use different approaches.

Min-max normalization:

It commonly maps a value to the interval :

• Produces a fixed range.
• Is sensitive to extreme values.
• Is useful for neural networks and algorithms requiring bounded inputs.

Standardization:

It transforms a feature to have approximately zero mean and unit variance:

• Does not produce a fixed range.
• Is often suitable for logistic regression, support vector machines, principal component analysis, and K-means.
• Is generally less affected by the original measurement units.

Both techniques prevent large-scale features from dominating distance calculations or parameter estimation. Their parameters must be calculated from the training set and reused for validation, testing, and future observations.

Categorical variables must often be converted into numerical representations before model training.

Major encoding techniques include:

• Label encoding: Assigns an integer to each category. It is appropriate for genuinely ordered categories but may create a false order for nominal data.
• Ordinal encoding: Maps ordered categories to values that preserve their ranking, such as low , medium , and high .
• One-hot encoding: Creates one binary feature for each category. It avoids artificial ordering but can create many columns.
• Frequency encoding: Replaces each category with its occurrence count or proportion.
• Target encoding: Replaces a category with a statistic derived from the target, such as the category mean. It requires careful cross-validation to avoid leakage.
• Binary encoding: Converts category codes into binary digits and is useful for features with many categories.

The method should be selected according to category order, cardinality, model type, available data, and leakage risk. Unknown categories that appear after deployment must also be handled explicitly.

A dataset is divided into separate subsets to estimate how well a predictive model will perform on unseen data.

• Training set: Used to learn model parameters and patterns.
• Validation set: Used to compare models, select features, and tune hyperparameters.
• Test set: Used only after model development to provide a final, unbiased performance estimate.

A typical division may be training, validation, and testing, although the appropriate proportions depend on dataset size.

Important considerations:

• Classification splits may be stratified to preserve class proportions.
• Time-series data should be split chronologically rather than randomly.
• Duplicate or related observations should not appear across subsets.
• Preprocessing parameters must be fitted using only the training set.

Using test data during model selection causes data leakage and produces an overly optimistic estimate of model performance.

Data leakage occurs when information unavailable at prediction time, or information from validation or test data, is used during model training. The model then appears more accurate than it will be in real use.

Common causes include:

• Calculating scaling or imputation values using the complete dataset.
• Selecting features using test-set results.
• Including a feature created after the target event occurred.
• Placing duplicate or closely related records in different data subsets.
• Applying target encoding without proper cross-validation.
• Randomly splitting time-series data and allowing future information into training.

Consequences:

• Inflated validation and test scores
• Poor performance after deployment
• Incorrect feature importance
• Misleading business decisions

Prevention:

• Split data before fitting preprocessing operations.
• Use pipelines to keep preprocessing and modeling together.
• Fit transformations only on training folds.
• Respect chronological and group boundaries.
• Audit whether every feature would be available at prediction time.

Preventing leakage is essential for obtaining a realistic estimate of generalization performance.

Feature selection is the process of choosing the most relevant input variables for constructing a predictive model.

Importance of feature selection:

• Reduces model complexity and training time.
• Removes irrelevant and redundant variables.
• May improve generalization and reduce overfitting.
• Makes models easier to interpret.
• Reduces data collection and storage costs.
• Can reduce problems caused by highly correlated variables.

Main approaches:

• Filter methods: Rank features using statistics such as correlation, chi-square score, mutual information, or variance.
• Wrapper methods: Evaluate subsets by repeatedly training a model, as in recursive feature elimination.
• Embedded methods: Select features during model training, as in Lasso regression or tree-based importance.

Feature selection must be performed using training data only. It should also consider domain knowledge because a statistically weak feature may still be important for fairness, control, or interpretation.

Feature selection and feature extraction both reduce input complexity, but they operate differently.

Feature selection examples:

• Removing constant columns
• Selecting variables with high mutual information
• Using Lasso to eliminate weak predictors

Feature extraction examples:

• Principal component analysis
• Text embeddings
• Combining date fields into elapsed time
• Extracting edges or textures from images

Feature selection is preferred when interpretability is important, while feature extraction is useful when many variables are correlated or when raw data must be converted into a compact representation.

Duplicate, inconsistent, and noisy data reduce dataset quality and may bias a predictive model.

Duplicate data:

• Identify exact duplicates by comparing complete rows or unique identifiers.
• Detect near-duplicates using record linkage, fuzzy matching, or similarity measures.
• Retain the authoritative or most recent record when duplicates conflict.

Inconsistent data:

• Standardize text case, spelling, units, and date formats.
• Map equivalent values such as "M", "Male", and "male" to one representation.
• Validate values using permitted ranges and reference tables.
• Enforce logical constraints, such as an end date not preceding a start date.

Noisy data:

• Use smoothing, aggregation, or binning when appropriate.
• Investigate unusual values with statistical and visual methods.
• Correct known measurement errors.
• Use robust models when noise cannot be removed safely.

Every cleaning rule should be documented and reproducible. Records should not be removed merely because they are unusual; their validity and relevance must first be examined.

Overfitting occurs when a model learns the training data too closely, including noise and accidental patterns. It achieves low training error but high validation or test error.

Methods to reduce overfitting:

• Collect more representative training data.
• Reduce model complexity.
• Apply regularization.
• Perform feature selection.
• Use cross-validation.
• Prune decision trees or use early stopping.
• Apply data augmentation where appropriate.

Underfitting occurs when a model is too simple to capture important patterns. It performs poorly on both training and unseen data.

Methods to reduce underfitting:

• Use a more flexible model.
• Add informative features.
• Reduce excessive regularization.
• Train the model for longer when optimization is incomplete.
• Improve feature transformations.

A suitable model achieves a balance between fitting the training data and generalizing to unseen observations. This balance is commonly described through the bias-variance trade-off.

The bias-variance trade-off explains how model complexity affects predictive error.

• Bias is error caused by overly simple assumptions. A high-bias model may miss important relationships and underfit the data.
• Variance is error caused by excessive sensitivity to the training sample. A high-variance model may learn noise and overfit.
• Irreducible error is random variation that cannot be eliminated using the available predictors.

The expected prediction error can be expressed conceptually as:

As model complexity increases, bias generally decreases while variance generally increases. The objective is to choose the complexity that minimizes validation error rather than training error alone.

Cross-validation, regularization, feature selection, and ensemble methods can help achieve an appropriate balance between bias and variance.

Class imbalance occurs when one target class contains substantially more observations than another. For example, fraudulent transactions may form less than of all transactions.

Effects on learning:

• A model may favor the majority class.
• Overall accuracy may appear high even when minority cases are rarely detected.
• Decision boundaries may not adequately represent the minority class.
• Important events such as fraud or disease may be missed.

Handling methods:

• Random undersampling: Remove some majority-class observations.
• Random oversampling: Duplicate minority-class observations.
• Synthetic sampling: Generate minority examples using methods such as SMOTE.
• Class weighting: Assign a larger penalty to mistakes on the minority class.
• Threshold adjustment: Change the classification threshold according to costs and objectives.
• Appropriate metrics: Use precision, recall, -score, area under the precision-recall curve, and the confusion matrix instead of accuracy alone.

Resampling should be performed only on the training data so that validation and test sets preserve realistic class distributions.

A complete preprocessing plan for customer churn prediction may include the following steps:

1. Define the target: Specify churn using a fixed observation window and prediction horizon.
2. Audit the data: Examine schemas, data types, ranges, class proportions, missing values, duplicates, and data sources.
3. Prevent leakage: Exclude information recorded after churn, such as account-closure reasons or final settlement details.
4. Split the data: Create training, validation, and test sets. Use chronological splitting if deployment predicts future churn.
5. Clean records: Remove true duplicates, correct invalid dates, standardize category labels, and validate numerical ranges.
6. Handle missing values: Use median imputation for skewed numerical fields, mode or explicit unknown categories for categorical fields, and missingness indicators where useful.
7. Treat outliers: Investigate extreme charges, tenure, and usage values; correct errors and cap only when justified.
8. Encode categories: Apply one-hot encoding to nominal fields and ordinal encoding to ordered fields.
9. Scale numerical features: Standardize variables for scale-sensitive algorithms.
10. Engineer features: Create tenure groups, recent usage changes, complaint frequency, and payment-failure rates.
11. Handle imbalance: Use class weights or training-set resampling if churn is rare.
12. Build a pipeline: Fit every transformation only on training data and apply the same learned transformations to new data.
13. Validate quality: Check transformed distributions, unknown categories, leakage, and metric stability across customer groups.

This plan produces reproducible model inputs while preserving a realistic estimate of performance on future customers.