Unit 2: Data Preparation and Machine Learning Workflow

I. Orientation

Data preparation converts raw observations into a reliable numerical form that a machine-learning algorithm can analyze. The workflow normally moves from collection and inspection to cleaning, transformation, feature construction, model training, and validation. The central principle is that a model can only learn useful patterns when the input data are representative, consistent, and processed without allowing test information to leak into training.

• Data as evidence: A dataset contains observations, features, and often a target variable; for example, house records may contain area, location, age, and sale price.
• Rows and columns: A row usually represents one observation, while a column represents one measured attribute or feature.
• Learning objective: Supervised learning estimates a relationship between input features (X) and a target (y); unsupervised learning searches for structure without a target.
• Reproducibility: The same cleaning rules, transformations, and random seed should produce comparable results.
• Data leakage control: Information calculated from validation or test data must not influence training decisions.
• Workflow convention: Exploration informs preprocessing, preprocessing supports feature engineering, and validation estimates performance on unseen data.

II. Introduction to NumPy, Pandas and Matplotlib

These three Python libraries form a practical foundation for numerical computation, tabular data handling, and visualization in machine learning. NumPy supplies efficient arrays, Pandas organizes labelled datasets, and Matplotlib displays patterns that may be hidden in raw tables.

A. Introduction to NumPy, Pandas and Matplotlib

The purpose of this toolkit is to move efficiently from numerical data to interpretable analysis.

• NumPy arrays: A NumPy array stores values in a compact, usually homogeneous structure.

import numpy as np

x = np.array([2, 4, 6, 8])
mean_x = np.mean(x)
scaled_x = (x - mean_x) / np.std(x)

• Symbols: x is the array, mean_x is its arithmetic mean, and np.std(x) is its standard deviation.
• Vectorization: (x - mean_x) operates on every element without an explicit loop, improving readability and speed.
  • Array dimensions: shape describes dimensions; a matrix with 100 rows and 5 columns has shape (100, 5).
  • Pandas Series: A Series is a labelled one-dimensional sequence, such as a single age column.
  • Pandas DataFrame: A DataFrame is a two-dimensional labelled table suitable for mixed column types.

import pandas as pd

df = pd.read_csv("customers.csv")
df.info()
df.describe()

• Concrete operations: read_csv loads comma-separated records, info() reports types and missingness, and describe() summarizes numerical columns.
  • Matplotlib plots: Matplotlib creates figures such as histograms, scatter plots, and line graphs.

import matplotlib.pyplot as plt

plt.hist(df["age"], bins=10)
plt.xlabel("Age")
plt.ylabel("Frequency")
plt.show()

• Interpretation: bins=10 divides observed ages into ten intervals; the plot can reveal skewness or unusual concentrations.

III. Understanding Data

Understanding Data establishes what each observation means, how it was obtained, and what patterns or errors are present before modeling begins. Sound analysis depends on domain context as well as numerical summaries.

A. Types of Datasets

The type of dataset determines appropriate algorithms, transformations, and evaluation methods.

• Tabular data: Rows and columns represent structured records, such as 10,000 patients with age, blood pressure, and diagnosis.
• Numerical data: Continuous values, such as temperature (23.6^\circ C), and discrete counts, such as number of purchases, support arithmetic operations.
• Categorical data: Labels such as red, blue, or premium represent groups rather than measured quantities.
  • Nominal categories: Categories have no natural order, such as blood type.
  • Ordinal categories: Categories have an order, such as low, medium, and high.
• Time-series data: Observations are ordered by time; stock prices at 09:30 and 09:31 cannot be treated as randomly interchangeable.
• Text, image, and audio data: These unstructured formats require transformations such as token counts, pixel arrays, or spectrograms before conventional models can use them.
• Labels and targets: In supervised learning, (X) denotes input features and (y) denotes the output to predict. Regression targets are numerical; classification targets are class labels.

B. Data Collection

Data Collection determines whether the dataset represents the real problem and its intended population.

• Source definition: Records may come from sensors, surveys, databases, transaction systems, application logs, or public repositories.
• Sampling frame: The frame identifies who or what could be selected; a model trained only on urban customers may generalize poorly to rural customers.
• Sampling bias: If one group is systematically overrepresented, learned patterns may reflect collection practices rather than reality.
• Measurement quality: A sensor recorded in meters must not be combined with a centimeter column without conversion; (1\text{ m}=100\text{ cm}).
• Data documentation: Record feature meanings, units, collection dates, inclusion rules, and the target definition.
• Privacy and consent: Personal identifiers should be minimized, protected, or removed when they are not necessary for prediction.
• Duplicate control: Repeated copies of one customer or event can make performance appear better because nearly identical records occur in both training and test sets.

C. Exploratory Data Analysis

Exploratory Data Analysis (EDA) uses summaries and visualizations to discover distributions, relationships, errors, and possible modeling choices.

• Shape and schema: df.shape gives row and column counts; df.dtypes identifies numerical, categorical, datetime, or object columns.
• Univariate inspection: Histograms show one variable’s distribution; a long right tail in income may suggest a logarithmic transformation.
• Relationship inspection: A scatter plot of house area against price can reveal correlation, nonlinearity, clusters, or heteroscedasticity.
• Group comparison: Grouping sales by region can expose different averages and unequal sample sizes.
• Summary statistics: Mean is sensitive to extreme values, while the median is more robust; for values (2, 3, 4, 100), the mean is (27.25), but the median is (3.5).
• Target balance: In classification, count each class; 950 negative and 50 positive cases indicate a 95:5 imbalance that makes accuracy potentially misleading.

IV. Data Preprocessing

Data Preprocessing converts imperfect raw data into consistent inputs while preserving valid information. Every operation should be fitted on training data and then applied to validation or test data.

A. Handling missing values

Handling missing values prevents algorithms from failing or treating absence as an unexamined numerical value.

• Missingness diagnosis: Use df.isna().sum() to count missing entries by column and compare missingness across target classes or collection sources.
• Deletion: Remove a row when only a few records are incomplete and their removal is unlikely to bias the sample; remove a column when most values are absent and the feature has little importance.
• Numerical imputation: Replace a missing numerical value with the training median, which is less affected by outliers than the mean.

median_age = train["age"].median()
train["age"] = train["age"].fillna(median_age)
valid["age"] = valid["age"].fillna(median_age)

• Leakage rule: The median is calculated from train, not from the combined dataset or valid.
  • Categorical imputation: Use the most frequent category or an explicit label such as Unknown when absence itself may carry meaning.
  • Model-based methods: K-nearest-neighbor or iterative imputation can estimate values from other features, but they add assumptions and computational cost.
  • Missingness indicator: Add (m=1) when a value was missing and (m=0) otherwise if the fact of missingness may predict the target.

B. Detecting outliers

Detecting outliers identifies observations that are unusual, erroneous, or genuinely important.

• Domain limits: A human age of (-4) is invalid, while an age of 104 may be rare but plausible; domain rules should precede automatic deletion.
• Interquartile range: Let (Q_1) and (Q_3) be the first and third quartiles, and (IQR=Q_3-Q_1). Flag values below (Q_1-1.5IQR) or above (Q_3+1.5IQR).
• Z-score: For mean (\mu) and standard deviation (\sigma), (z=(x-\mu)/\sigma). A common screening threshold is (|z|>3), assuming the distribution is reasonably suitable for this rule.
• Visual detection: Box plots, scatter plots, and histograms can show isolated points or measurement clusters.
• Treatment choices: Correct impossible values, cap extreme values, transform skewed variables, use robust statistics, or remove records only with documented justification.
• Model sensitivity: Linear regression and distance-based algorithms are often more affected by extreme magnitudes than tree-based models.

V. Feature Engineering

Feature Engineering creates or transforms model inputs so that relevant structure is easier for an algorithm to learn. Good features encode domain meaning without using information unavailable at prediction time.

A. Introduction to feature scaling

Feature scaling puts numerical variables on comparable numerical ranges.

• Standardization: Transform (x) into (z=(x-\mu)/\sigma), where (\mu) is the training mean and (\sigma) is the training standard deviation; the resulting feature has approximately mean 0 and standard deviation 1.
• Min-max scaling: Transform (x) into (x'=(x-x{\min})/(x{\max}-x_{\min})), usually producing values between 0 and 1.
• When needed: K-nearest neighbors, support vector machines, neural networks, and gradient-based models use distances or coefficient magnitudes and therefore usually benefit from scaling.
• When less important: Decision trees split by thresholds and generally do not require comparable feature scales.
• Leakage prevention: Fit (\mu,\sigma), or minimum and maximum values using training data, then reuse those values for validation and test data.

B. Feature encoding

Feature encoding converts categorical information into numerical representations without inventing misleading relationships.

• One-hot encoding: A category column with red, blue, and green becomes binary columns such as color_red, color_blue, and color_green.
• Ordinal encoding: Ordered categories can be mapped deliberately, for example low=0, medium=1, and high=2; this is inappropriate for unordered categories.
• Binary encoding: Two-category values such as yes and no can become (1) and (0).
• High-cardinality caution: Encoding thousands of unique postal codes may create sparse, overfit features; grouping, frequency encoding, or domain-based representations may be preferable.
• Unknown categories: Production data may contain a category absent during training, so encoders should provide an “unknown” handling strategy.

C. Feature selection

Feature selection keeps informative inputs and removes irrelevant, redundant, or unsafe variables.

• Filter methods: Correlation, variance thresholds, and statistical tests rank features before model training; a constant column has zero variance and provides no discrimination.
• Wrapper methods: Recursive feature elimination repeatedly trains a model and removes the least useful features, but can be computationally expensive.
• Embedded methods: Lasso regression uses an (L_1) penalty, shrinking some coefficients to zero; the objective includes (\lambda\sum_j|w_j|), where (w_j) is a feature weight and (\lambda) controls penalty strength.
• Leakage screening: Remove identifiers, future outcomes, or variables created after the prediction event.
• Validation requirement: Select features using training folds only; selecting them from the full dataset can make evaluation optimistically biased.

VI. Training and Validation

Training and Validation measure whether a learned pattern generalizes beyond the observations used to fit the model. The dataset is partitioned so that evaluation data remain unseen during fitting and selection.

A. Train-Test Split

A Train-Test Split creates separate data for model development and final performance estimation.

• Training set: The model learns parameters from the training subset, such as regression coefficients or tree thresholds.
• Test set: The test subset is held back until preprocessing choices, features, and hyperparameters are finalized.
• Typical proportion: An 80:20 split gives 80% of observations to training and 20% to testing, although the appropriate ratio depends on dataset size.
• Random state: A fixed seed makes a random split reproducible.

from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.20, random_state=42, stratify=y
)

• Symbols: X contains features, y contains targets, test_size=0.20 reserves 20%, and stratify=y preserves class proportions.
  • Time ordering: Time-series data should generally use chronological splits, because future observations must not train a model evaluated on the past.
  • Evaluation metrics: Classification may use precision, recall, F1-score, or ROC-AUC; regression may use MAE, MSE, or (R^2). Metric choice should reflect the practical cost of errors.

B. Cross Validation

Cross Validation repeatedly trains and evaluates a model on different partitions to obtain a more stable estimate during model selection.

• K-fold procedure: Divide data into (K) folds; train on (K-1) folds and validate on the remaining fold, repeating until every fold has served as validation data.
• Mean score: If validation scores are (s_1,\ldots,sK), the estimate is (\bar{s}=\frac{1}{K}\sum{i=1}^{K}s_i); variation across scores indicates sensitivity to the split.
• Stratified K-fold: For classification, preserve class proportions in each fold, especially when the minority class is small.
• Nested processing: Imputation, scaling, encoding, and feature selection must occur inside each training fold, commonly through a pipeline.
• Final test discipline: Use cross-validation to select the model and hyperparameters, then evaluate the chosen process once on the untouched test set.
• Limitations: Cross-validation costs approximately (K) training runs and can be unreliable when observations are dependent, as in grouped or time-ordered data.