Unit 2: Data Pre-processing

I. Foundations of Data Preparation

Data preprocessing is the systematic transformation of raw data into a reliable, consistent, and analysis-ready form. It precedes statistical analysis and machine learning because model quality depends not only on the algorithm but also on the validity, representation, and relevance of its input data.

A. Introduction to data preprocessing

Data preprocessing establishes the quality and structure required for trustworthy data-driven conclusions.

• Governing principle — “garbage in, garbage out”: Incorrect, incomplete, or poorly represented inputs produce misleading outputs even when the analytical method is mathematically correct.
• Primary objective: Convert observations from their collected state into a dataset whose rows, columns, values, and metadata have clear and consistent meanings.
• Core characteristics:
  • Validity: Values obey defined rules; for example, an examination score must lie between 0 and 100.
  • Accuracy: Values represent real conditions; a recorded temperature of 220°C for a patient is structurally numeric but factually inaccurate.
  • Completeness: Required fields are populated or their absence is explicitly represented.
  • Consistency: Equivalent facts use equivalent formats, units, and labels, such as storing all masses in kilograms.
  • Uniqueness: Each real-world entity or event is represented only as intended.
  • Timeliness: Data belongs to the period relevant to the analysis.
• Typical input sources: Transaction databases, spreadsheets, application logs, sensors, surveys, web services, images, and documents produce data with different structures and quality risks.
• Analytical dependence: Preprocessing choices must reflect the intended task. Scaling may be essential for distance-based clustering but unnecessary for a decision tree.
• Reproducibility convention: Each operation should be recorded as code or pipeline metadata rather than performed through undocumented manual editing.
• Data leakage rule: Information unavailable at prediction time must not influence model training. Imputation and scaling parameters should therefore be learned from training data only.

II. Data Preprocessing — Building Analysis-Ready Data

Data preprocessing is the broader technical process of inspecting, cleaning, integrating, transforming, reducing, and validating data before analysis.

A. Data preprocessing

The process applies ordered quality controls and transformations while preserving the meaning of the original observations.

• 1. Data profiling: Examine dimensions, column types, ranges, frequencies, missingness, and relationships before changing values.
  • A table with 10,000 rows and 12 columns should be checked for unexpected row loss, constant columns, and impossible category labels.
• 2. Data cleaning: Correct or manage missing, duplicate, inconsistent, noisy, and invalid observations.
  • Missing numerical values may be imputed with a median when a skewed distribution makes the mean unrepresentative.
  • Missingness may itself be encoded by an indicator such as income_missing = 1.
  • Exact duplicate rows can be removed only after confirming that repeated events are not legitimate observations.
• 3. Data integration: Combine sources through keys, joins, or record linkage.
  • Joining an orders table to a customers table through customer_id requires checking key uniqueness and unmatched records.
  • A many-to-many join can multiply rows unexpectedly, changing totals and distributions.
• 4. Data transformation: Change representation without changing the underlying concept.
  • Categorical labels may be one-hot encoded into binary columns.
  • Dates may yield features such as year, month, weekday, or elapsed duration.
  • Skewed positive values may use a logarithmic transformation, subject to domain and zero-value constraints.
• 5. Feature scaling: Place numerical variables on comparable scales where an algorithm depends on magnitude.
  • Min–max normalization maps a value to a chosen interval, commonly [0,1]:

x' = (x - xmin) / (xmax - xmin)

Here, x is the original value, x' is the scaled value, and xmin and xmax are the minimum and maximum learned from the training set.

• Standardization uses z = (x - μ) / σ, where μ is the training mean and σ is the training standard deviation.
  • 6. Data reduction: Decrease size or dimensionality while retaining useful information.
• Methods include sampling, aggregation, feature selection, binning, and principal component analysis.
• Reduction lowers storage and computation but may discard rare or subtle patterns.
  • 7. Validation: Recalculate row counts, uniqueness, ranges, missing-value rates, and aggregate totals after processing.

B. Applications and limitations

Preprocessing improves analytical reliability, but every transformation introduces assumptions that must be controlled.

• Applications: It supports dashboards, statistical inference, forecasting, classification, clustering, recommendation, and anomaly detection.
• Pipeline consistency: The same fitted transformations must be applied to validation, test, and future production records.
• Information-loss limitation: Removing outliers or collapsing categories can erase valid minority cases.
• Bias limitation: Imputation based on historically biased observations can preserve or intensify existing disparities.
• Audit requirement: Raw data should remain immutable, with processed versions linked to transformation code, parameters, and timestamps.

III. Data Wrangling — Reshaping and Combining Data

Data wrangling is the practical manipulation of data into a structure suitable for a particular analysis; it overlaps with preprocessing but emphasizes organization, restructuring, and integration.

A. Data wrangling

Wrangling turns inconvenient tables and files into coherent datasets whose observational units and variables are explicit.

• Tidy-data principle:
  • Each variable occupies one column.
  • Each observation occupies one row.
  • Each type of observational unit occupies one table.
  • For example, monthly sales stored in columns Jan, Feb, and Mar can be reshaped into columns month and sales.
• Selection and filtering: Retain relevant variables and observations using explicit conditions, such as selecting orders where status == "completed".
• Reshaping:
  1. Wide-to-long transformation: Converts repeated measurement columns into variable–value pairs; this is useful for plotting time series.
  2. Long-to-wide transformation: Spreads category values into separate columns; this is useful for matrices and reports.
• Parsing: Split compound values into meaningful fields. The value 2026-09-10 can be parsed into year 2026, month 09, and day 10.
• Aggregation: Summarize lower-level observations using counts, sums, means, minima, or maxima.
  • Daily transactions may be aggregated to monthly revenue, but transaction-level variation is then unavailable.
• Joining:
  • Inner join: Retains matching keys from both tables.
  • Left join: Retains every row from the left table and inserts missing values where no right-side match exists.
  • Full join: Retains matched and unmatched keys from both tables.
• String normalization: Trim whitespace, standardize case, and map equivalent labels; "New York", "new york", and " New York " should not accidentally form three categories.
• Wrangling workflow:

load → inspect → select → clean → reshape → join → validate → save

Each arrow denotes a controlled transition whose row counts, keys, and schema should be checked.

B. Applications and limitations

Wrangling enables datasets to fit analytical tools, although structural convenience must not override semantic correctness.

• Applications: Common uses include consolidating survey files, creating time-series panels, merging customer histories, and converting logs into event tables.
• Granularity risk: Joining customer-level data to order-level data repeats customer attributes across orders; aggregation must account for that repetition.
• Schema risk: Source columns may change names or types, causing a previously valid pipeline to fail or silently mis-handle data.
• Documentation need: Renamed columns, derived variables, join rules, and excluded records should be stored with the resulting dataset.

IV. Data Types and Forms — Representing Observations Correctly

A data type defines what a value represents and which operations are meaningful, while a data form describes how values are structurally organized.

A. Data types and forms

Correct classification prevents invalid calculations and guides storage, visualization, transformation, and modeling choices.

• Qualitative data:
  1. Nominal: Categories have no inherent order, such as blood group {A, B, AB, O}; computing their arithmetic mean is meaningless.
  2. Ordinal: Categories have an order but not necessarily equal intervals, such as {low, medium, high}.
  3. Binary: Exactly two states, commonly encoded as 0 and 1; the semantic meaning of each code must be documented.
• Quantitative data:
  1. Discrete: Countable values, such as the number of purchases 0, 1, 2, ....
  2. Continuous: Measurements within an interval, such as height 172.4 cm.
  3. Interval scale: Equal differences are meaningful, but zero is not absolute; Celsius temperature is an example.
  4. Ratio scale: Equal differences and ratios are meaningful because zero represents absence; mass in kilograms is an example.
• Common computational types: Integers, floating-point numbers, Boolean values, strings, dates, timestamps, and categorical codes determine permitted software operations.
• Structured data: Relational tables have predefined rows, columns, keys, and schemas.
• Semi-structured data: JSON, XML, and event logs contain labels or nesting without a fixed rectangular table.
• Unstructured data: Text, audio, images, and video require feature extraction before most statistical methods can use them.
• Cross-sectional form: Many entities are measured at one time, such as one income record per household in 2026.
• Time-series form: One or more variables are ordered through time, such as hourly electricity demand.
• Panel form: Multiple entities are repeatedly measured, such as annual income for each household from 2020–2026.
• Type-conversion risk: An identifier such as ZIP code 02139 is nominal text, not a quantity; integer conversion would remove the leading zero and imply invalid arithmetic.

V. Possible Data Error Types — Detecting Quality Failures

Data errors are deviations between recorded data and the intended real-world value, definition, structure, or constraint.

A. Possible data error types

Recognizing error classes helps determine whether values should be corrected, excluded, imputed, flagged, or investigated.

• Missing-data errors: Values may be absent because of nonresponse, device failure, optional fields, or failed joins.
  • Missing completely at random, missing at random, and missing not at random imply different risks for estimation.
• Validity errors: A value violates a domain rule, such as age −4, probability 1.3, or February 30.
• Range errors: Values fall outside an accepted interval even if their storage type is correct.
• Type and format errors: A numeric column may contain "unknown", or dates may mix DD/MM/YYYY with MM/DD/YYYY.
• Unit errors: Measurements recorded in metres and feet may be combined without conversion, creating plausible-looking but incomparable values.
• Transcription errors: Manual entry may transpose digits, turning 54 into 45, or place a decimal incorrectly, turning 7.5 into 75.
• Duplicate errors: The same entity or event may appear more than once because of repeated imports or inconsistent identifiers.
• Consistency errors: Related fields contradict each other; for example, employment_status = "unemployed" conflicts with a recorded employer name unless an exception is defined.
• Referential-integrity errors: A foreign key such as customer_id = 918 has no corresponding customer record.
• Outliers and noise: An outlier is unusually distant from other observations, while noise is random variation or measurement disturbance.
  • An outlier may be a genuine rare case rather than an error and should not be removed solely because it is extreme.
• Label errors: In supervised learning, an example may receive the wrong target class, directly teaching the model an incorrect relationship.
• Sampling and coverage errors: The dataset may exclude parts of the target population or overrepresent easily observed groups.
• Temporal errors: Stale records, inconsistent time zones, clock drift, or future information can invalidate chronological analysis.
• Detection controls: Schema validation, range checks, uniqueness constraints, frequency tables, cross-field rules, visual plots, and reconciliation against source totals expose different error classes.
• Treatment principle: Preserve the original value, record the reason for every correction, and distinguish confirmed errors from unusual but valid observations.