Unit 1: Data preprocessing and visualization

BTY587 — Data Analysis And Simulations 7 min read

Data preprocessing and visualization form the first stage of any analysis or simulation pipeline (the "garbage in, garbage out" principle): raw data must be typed, cleaned and explored before any model is fit. This unit establishes the vocabulary of data types, the mechanics of repairing missing values, four exploratory plots, and the reduction of high-dimensional data to a manageable feature space.

I. Foundations of the data pipeline

The analytic value of a dataset depends on knowing what each variable is before deciding what may be done to it. Everything downstream — the summary statistic, the plot, the distance metric — is licensed or forbidden by variable type and data quality.

Observation and feature: a dataset is an n×p matrix; rows are observations (records, samples), columns are features (variables, attributes).
Population vs sample: analysis infers about a population from a sample of size n; preprocessing decisions bias that inference if made carelessly.
Tidy structure: one variable per column, one observation per row, one value per cell — the precondition for most plotting and modelling libraries.
Measurement error vs missingness: a wrong value and an absent value are different faults, handled by validation and imputation respectively.

II. Types of data

Classification by measurement scale

Data type dictates permissible arithmetic and the correct visual encoding. Two broad families split into four scales.

1. Qualitative (categorical): labels with no inherent arithmetic.
- Nominal: unordered categories — e.g. blood group {A, B, AB, O}, country. Only counts and mode are valid.
- Ordinal: ordered but with unequal/unknown gaps — e.g. Likert {poor < fair < good}. Median and rank order are valid; differences are not.
2. Quantitative (numeric): measured magnitudes.
- Interval: ordered with equal gaps but arbitrary zero — e.g. temperature in °C; 20°C is not "twice" 10°C. Addition valid, ratios not.
- Ratio: equal gaps and true zero — e.g. mass in kg, count. All arithmetic including ratios valid.

Discrete vs continuous: discrete numeric takes isolated values (number of defects: 0, 1, 2…); continuous takes any value in an interval (height = 172.4 cm). This distinction chooses bar chart vs histogram.

Encoding for computation: nominal features are converted to numbers by one-hot encoding (k categories → k binary columns) so no false order is implied; ordinal features may use label encoding (poor=1, fair=2, good=3) where order is genuine.

III. Dealing with missing data

Detection, mechanism, and repair

Missing entries (coded NaN, NULL, or a sentinel like -999) must be diagnosed by mechanism before choosing a treatment, because the wrong choice biases estimates.

Missingness mechanisms:
- MCAR (Missing Completely At Random): probability of missing is independent of all data — e.g. a sample tube dropped by accident. Safe to delete.
- MAR (Missing At Random): missingness depends on observed variables — e.g. income unreported more often by younger respondents. Imputable from correlates.
- MNAR (Missing Not At Random): missingness depends on the unobserved value itself — e.g. high earners hiding income. Requires explicit modelling.
1. Deletion methods:
- Listwise deletion: drop any row with a missing cell; simple but wastes data and biases unless MCAR.
- Pairwise deletion: use all available cases per calculation; retains data but yields inconsistent covariance matrices.
2. Imputation methods:
- Mean/median/mode: replace with the column statistic; median resists outliers. Shrinks variance and distorts correlations.
- KNN imputation: fill from the k nearest complete rows by distance; preserves local structure.
- Regression / MICE: predict the missing feature from the others iteratively; principled under MAR.

PYTHON

import pandas as pd
df.isna().sum()                     # count missing per column
df['age'].fillna(df['age'].median(), inplace=True)   # median imputation

Missing-indicator flag: add a binary column marking imputed cells so a model can learn whether missingness itself carries signal.

IV. Exploratory visualization

Purpose and principle

Visualization exposes distribution, relationship and anomaly that summary numbers hide (Anscombe's quartet: four datasets sharing mean, variance and correlation yet looking utterly different). Each plot below is chosen by the number and type of variables displayed.

A. Scatter plot

Displays the joint relationship between two continuous variables as points at coordinates (xᵢ, yᵢ).

Reads: correlation direction (positive/negative), form (linear/curved), strength (tight/diffuse), and bivariate outliers.
Anchoring statistic: the Pearson coefficient r summarises linear strength.

TEXT

r = Σ(xᵢ−x̄)(yᵢ−ȳ) / sqrt(Σ(xᵢ−x̄)²·Σ(yᵢ−ȳ)²)

xᵢ, yᵢ: paired values; x̄, ȳ: their means; r ∈ [−1, 1].
- Enhancements: a third variable can be encoded by point colour (hue) or size (bubble chart); a trend line overlays the fitted relationship.
- Limitation: overplotting — with many points dense regions saturate; remedied by transparency (alpha), hexbin, or 2-D density.

B. Histogram

Shows the distribution of one continuous variable by binning its range and plotting frequency per bin as adjacent bars.

Construction: split range into k bins of width h; bar height = count (or density) in each bin. Bars touch, unlike a bar chart, signalling continuity.
Bin-width rule: Sturges k = 1 + log₂(n); or Freedman–Diaconis h = 2·IQR·n^(−1/3), which resists outliers.
Reads: modality (uni/bimodal), skew (long tail direction), and spread.
Worked example: ages {21, 22, 23, 25, 25, 26, 40}, bins of width 5 → [20–25): 3, [25–30): 3, [40–45): 1, revealing right skew and a gap.
Limitation: appearance is sensitive to bin count and origin; a kernel density estimate gives a smooth alternative.

C. Group plots

Compares the distribution of one variable across categories by drawing sub-panels or overlaid series per group.

Faceting (small multiples): one panel per category on shared axes — e.g. a histogram of sales per region, panels aligned so shapes compare directly.
Grouped bar / grouped box: categories placed side by side so a categorical × numeric relationship is read at a glance.
Design rule: keep axis scales identical across panels so differences reflect data, not rescaling; use consistent colour per group.
Use: answers "does the distribution shift with the category?" — the visual analogue of a group-by aggregation.

D. Box plots

Summarises a distribution through its five-number summary and flags outliers, ideal for comparing many groups compactly.

Five-number summary: minimum, Q1, median (Q2), Q3, maximum.
- Box: spans Q1 to Q3; its length is the interquartile range IQR = Q3 − Q1.
- Median line: inside the box; its offset shows skew.
- Whiskers: extend to the furthest point within 1.5·IQR of the box.
Outlier rule: points beyond Q1 − 1.5·IQR or Q3 + 1.5·IQR plotted individually.
Worked example: data with Q1 = 10, Q3 = 20 → IQR = 10; upper fence = 20 + 15 = 35, so a value of 40 is a flagged outlier.
Strength vs histogram: loses fine shape (cannot show bimodality) but stacks dozens of groups on one axis — a violin plot restores shape by adding a mirrored density.

V. Dimensionality reduction

Compressing many features into few

High-dimensional data suffers the curse of dimensionality — distances become uniform and models overfit — so p features are mapped to q ≪ p while retaining structure. Two aims: feature selection (keep a subset) and feature extraction (build new combinations).

1. Principal Component Analysis (PCA) — linear extraction:
- Principle: finds orthogonal axes (principal components) that maximise retained variance.
- Procedure: standardise features → compute covariance matrix Σ → eigen-decompose → keep the top q eigenvectors.