Unit 3: Various Data Preprocessing Operations

I. Orientation — Foundations of Data Preprocessing

Data preprocessing is the systematic conversion of raw data into a reliable, consistent, and analytically suitable form. It occurs before statistical analysis, visualization, or machine learning because real-world datasets commonly contain missing values, errors, incompatible formats, redundant features, and excessive detail.

• Governing principle: The quality of analytical output depends strongly on input quality—often summarized as “garbage in, garbage out.”
• Main objective: Preprocessing improves data quality while preserving information relevant to the intended analysis.
• Core dimensions of data quality:
  • Accuracy: Values correctly represent real-world entities or events.
  • Completeness: Required observations and attributes are present.
  • Consistency: Values do not contradict one another across records or sources.
  • Validity: Values satisfy permitted formats, types, ranges, and business rules.
  • Uniqueness: Each real-world entity is represented without unintended duplication.
  • Timeliness: Data is sufficiently current for its intended purpose.
• Typical workflow: Profiling identifies problems; cleaning corrects them; integration combines sources; transformation changes representation; reduction decreases volume; and discretization converts continuous attributes into intervals.
• Task dependence: A preprocessing decision must reflect the analytical objective. For example, removing an extreme transaction may help estimate normal spending but destroy evidence needed for fraud detection.
• Data leakage constraint: Parameters learned from data—such as means, medians, category mappings, or scaling factors—must be estimated from the training set and then applied unchanged to validation and test sets.
• Documentation requirement: Every operation should be reproducible through recorded rules, code, parameters, and data versions rather than undocumented manual edits.

II. Data Cleaning — Detecting and Correcting Data-Quality Problems

A. Data cleaning

Data cleaning identifies and handles incomplete, inaccurate, inconsistent, noisy, or duplicated observations so that the dataset satisfies defined quality requirements.

• Missing-value detection: Missingness may appear as NULL, NaN, empty strings, impossible sentinels such as -999, or phrases such as "unknown".
  • Deletion: Remove a row when missingness is rare and the observation contributes little; remove a feature when most of its values are absent and it is not essential.
  • Imputation: Replace missing numerical values with a mean or median and categorical values with a mode or explicit "Missing" category.
  • Model-based estimation: Predict absent values using regression, nearest neighbors, or multiple imputation when relationships among attributes are important.
• Missingness mechanisms:
  • MCAR: Missing Completely at Random; missingness is unrelated to observed or unobserved values.
  • MAR: Missing at Random; missingness depends on observed attributes.
  • MNAR: Missing Not at Random; missingness depends on the missing value itself, as when high earners decline to report income.
• Noise handling: Random measurement variation can be reduced through smoothing, binning, regression, or aggregation. Smoothing must not erase meaningful local patterns.
• Outlier detection: An observation may be flagged using domain limits, the interquartile range, or a standardized score. For a value (x),

z = (x - μ) / σ

• Symbols: (z) is the standardized score, (x) the observed value, (\mu) the mean, and (\sigma) the standard deviation.
• Interpretation: A large absolute (z)-score indicates distance from the mean, but it does not prove that the value is erroneous.
  • Inconsistency correction: Standardize representations such as "M", "Male", and "male" into one approved category; convert dates such as 10/09/2026 to an unambiguous standard like ISO 2026-09-10.
  • Duplicate removal: Exact duplicates match across selected fields; near-duplicates require record linkage using names, addresses, timestamps, or identifiers.
  • Rule validation: Constraints provide concrete checks—for example, quantity ≥ 0, end_date ≥ start_date, and each customer identifier must be unique.

B. Applications and limitations

Cleaning increases trust in analysis, but an automatic correction is safe only when supported by a valid rule.

• Applications: Clean records improve descriptive statistics, customer matching, model training, regulatory reporting, and dashboard accuracy.
• Worked example: For ages [19, 22, NULL, 24, 120], median imputation can replace NULL using valid ages, while 120 should be checked against source records rather than automatically deleted.
• Limitations: Imputation reduces apparent variability; deletion can introduce bias; and aggressive outlier removal can eliminate rare but genuine cases.
• Auditability: Preserve raw data separately and record the number of values modified, deleted, or imputed by each rule.

III. Data Integration — Combining Heterogeneous Sources

A. Data integration

Data integration combines information from multiple databases, files, applications, or services into a coherent dataset with aligned structures and meanings.

• Schema integration: Corresponding attributes must be identified despite naming differences, such as customer_id, cust_no, and client_key.
• Entity resolution: Records referring to the same real-world object are linked using stable identifiers or matching evidence. For example, two customer rows may match by normalized email and telephone number.
• Join operations:
  1. Inner join: Retains only keys appearing in both datasets.
  2. Left join: Retains every row from the left dataset and inserts missing values where the right dataset has no match.
• Key constraints: A primary key uniquely identifies a row, while a foreign key references a key in another table. Violations can create duplicated or unmatched records.
• Semantic conflicts: Identically named fields may have different meanings. One source may record revenue before tax while another records it after tax.
• Unit conflicts: Measurements must use compatible units. Temperature in Fahrenheit can be converted to Celsius by

C = (F - 32) × 5/9

• Symbols: (C) is temperature in degrees Celsius and (F) is temperature in degrees Fahrenheit.
  • Redundancy detection: Integration may introduce duplicate attributes or strongly correlated fields, such as annual_income and twelve summed monthly-income columns.
  • Provenance: Source system, extraction time, transformation rule, and record lineage should be retained to support auditing and error tracing.

B. Applications and limitations

Integration creates a broader analytical view, but incorrect matches can be more damaging than unmatched records.

• Applications: Organizations integrate sales, inventory, customer-support, and marketing data to construct unified customer or product views.
• Worked example: Joining an orders table with customers on customer_id enriches each order with region, but duplicated customer keys multiply order rows and inflate total revenue.
• Limitations: Sources may refresh at different times, apply incompatible definitions, or lack reliable identifiers.
• Validation: Compare row counts, unmatched-key rates, duplicate-key counts, and aggregate totals before and after each join.

IV. Data Transformation — Changing Representation or Scale

A. Data transformation

Data transformation converts values, structures, or formats into representations better suited to storage, comparison, statistical analysis, or machine learning.

• Normalization by min–max scaling: Values are mapped to a chosen interval, commonly ([0,1]).

x' = (x - xmin) / (xmax - xmin)

• Symbols: (x') is the scaled value, (x) the original value, and (x{\min}) and (x{\max}) the feature’s minimum and maximum.
  • Standardization: The (z)-score formula centers a feature near zero and scales it by its standard deviation. It is useful when variables such as income and age have very different numerical magnitudes.
  • Categorical encoding:
• One-hot encoding: Creates one binary column per nominal category, such as color_red and color_blue.
• Ordinal encoding: Assigns ordered values only when a genuine order exists, such as low < medium < high.
  • Mathematical transformation: Logarithmic transformation can reduce right skew in positive variables:

y = log(1 + x)

• Symbols: (x) is the original non-negative value and (y) its transformed value; adding 1 permits (x=0).
  • Aggregation: Detailed values are summarized, such as converting daily sales into monthly totals using SUM(amount) grouped by month.
  • Feature construction: Existing attributes generate informative features; for example, trip_duration = arrival_time - departure_time.
  • Generalization: Detailed categories may be mapped upward in a hierarchy, such as city → state → country.

B. Applications and limitations

Transformation can expose patterns and satisfy model assumptions, but it changes how values must be interpreted.

• Applications: Distance-based algorithms benefit from scaling because a large-unit feature would otherwise dominate Euclidean distance.
• Worked example: If a feature ranges from 20 to 100, min–max scaling maps (x=60) to (60−20)/(100−20)=0.5.
• Limitations: Min–max scaling is sensitive to extreme values, one-hot encoding can create thousands of columns, and a log-transformed coefficient is not interpreted on the original scale.
• Leakage prevention: Fit scaling bounds, means, encoding vocabularies, and transformation parameters only on training data.

V. Data Reduction — Decreasing Volume While Preserving Information

A. Data reduction

Data reduction produces a smaller representation of a dataset while retaining the information required for analysis.

• Dimensionality reduction: Reduces the number of attributes.
  1. Feature selection: Retains a subset of original variables using domain judgment, correlation, statistical tests, or model-based importance.
  2. Feature extraction: Constructs new variables from existing ones; Principal Component Analysis (PCA) creates orthogonal components that capture decreasing amounts of variance.
• Numerosity reduction: Replaces many observations with a smaller representation through sampling, clustering prototypes, regression models, or histograms.
• Sampling methods:
  • Simple random sampling: Gives each record an equal selection probability.
  • Stratified sampling: Samples separately within groups, helping preserve rare classes such as fraudulent transactions.
• Aggregation: Replaces fine-grained records with summaries—for example, one million sensor readings may become hourly mean, minimum, and maximum values.
• Compression: Lossless compression permits exact reconstruction; lossy compression sacrifices detail for a larger size reduction.
• PCA criterion: The proportion of variance retained by the first (k) components is

retained variance = (λ1 + ... + λk) / (λ1 + ... + λp)

• Symbols: (\lambda_i) is the variance represented by component (i), (k) is the number retained, and (p) is the total number of components.

B. Applications and limitations

Reduction lowers computational and storage costs but may discard rare or interpretable information.

• Applications: Reduced data accelerates visualization, clustering, model training, transmission, and exploratory analysis.
• Worked example: Retaining components with eigenvalues [5, 3] from a total variance of 10 preserves (5+3)/10 = 80% of variance.
• Limitations: PCA components combine original features and can be difficult to explain; unrepresentative samples introduce bias; aggregation hides individual events.
• Selection principle: Evaluate reduced data by downstream accuracy, variance retained, runtime saved, and whether minority groups remain adequately represented.

VI. Data Discretization — Converting Continuous Values into Intervals

A. Data discretization

Data discretization replaces continuous numerical values with a finite set of intervals or labels, simplifying patterns and enabling methods that require categorical input.

• Equal-width binning: Divides the range into (k) intervals of identical width.

width = (xmax - xmin) / k

• Symbols: (x{\max}) and (x{\min}) are the largest and smallest values, and (k) is the number of bins.
  • Equal-frequency binning: Places approximately the same number of observations in each bin; interval widths may differ substantially.
  • Supervised discretization: Uses a target variable to choose cut points that improve class separation, as in entropy-based decision-tree splits.
  • Unsupervised discretization: Uses only the feature distribution through equal-width, quantile, histogram, or clustering methods.
  • Domain-based intervals: Subject knowledge defines meaningful boundaries, such as age groups 0–17, 18–64, and 65+.
  • Boundary conventions: Intervals must explicitly state inclusion rules, such as [0, 10) for values greater than or equal to 0 but less than 10.
  • Concept hierarchies: Discretization may create several abstraction levels, such as exact age → age band → life-stage category.

B. Applications and limitations

Discretization improves simplicity and robustness when interval membership matters more than small numerical differences.

• Applications: It supports rule-based systems, frequency tables, categorical association analysis, interpretable risk bands, and some classification algorithms.
• Worked example: For values from 0 to 100 with (k=5), equal-width binning gives width 20 and intervals [0,20), [20,40), [40,60), [60,80), and [80,100].
• Advantages: Bins can reduce sensitivity to minor noise, expose non-linear patterns, and make model outputs easier to communicate.
• Limitations: Values within one interval become indistinguishable, boundary-adjacent values may be treated as different, and poorly selected cut points cause information loss.
• Evaluation: Effective bins should contain enough observations, preserve useful target relationships, remain stable across datasets, and have defensible labels and boundaries.