Unit 3: Various Data Preprocessing Operations - Subjective Questions

Data cleaning is the process of detecting and correcting inaccurate, incomplete, inconsistent, duplicated, or irrelevant data before analysis.

Major data-cleaning tasks include:

• Handling missing values: Fill missing values using a constant, mean, median, mode, regression, or another estimation technique.
• Smoothing noisy data: Reduce random errors using binning, regression, or clustering.
• Detecting outliers: Identify observations that differ significantly from normal patterns.
• Resolving inconsistencies: Standardize conflicting formats, units, labels, and business rules.
• Removing duplicates: Detect multiple records representing the same entity.
• Validating data: Check ranges, types, constraints, and relationships between attributes.

Effective data cleaning improves the accuracy, reliability, and interpretability of analytical results.

Missing data can be handled using the following techniques:

• Delete the record: Appropriate when very few records contain missing values and deletion does not introduce bias.
• Delete the attribute: Suitable when an attribute has too many missing values and is not essential.
• Manual filling: A domain expert supplies the value; accurate but costly for large datasets.
• Global constant: Replace missing values with a label such as Unknown. This is useful for categorical data but may create an artificial category.
• Mean, median, or mode: Mean is suitable for symmetric numerical data, median for skewed data, and mode for categorical data.
• Class-conditional mean: Replace a value using the mean of records belonging to the same class.
• Predictive imputation: Estimate values using regression, decision trees, nearest neighbors, or other models.

The selected method should consider the proportion of missing data and whether values are missing completely at random, at random, or not at random.

Noisy data contains random errors, incorrect measurements, or unexplained variation that obscures the underlying pattern.

Noise-reduction methods include:

• Binning: Sort values and divide them into intervals. Values in each bin are replaced by the bin mean, median, or nearest boundary.
• Regression: Fit a function such as and replace noisy observations with values estimated from the fitted relationship.
• Clustering: Group similar observations. Records far from all clusters or located in very small clusters may be treated as outliers.

For example, the sorted values can be divided into two bins. Smoothing by bin means replaces the first bin with and the second with .

Binning is simple, regression preserves trends, and clustering is useful when the data naturally forms groups.

These quality problems require different detection and treatment methods:

• Duplicate records: Detected through exact matching or approximate matching of identifiers, names, addresses, and other attributes. Duplicates may be deleted or merged into a single consolidated record.
• Inconsistent values: Found using type checks, range checks, format rules, integrity constraints, and cross-field validation. They are corrected by standardizing representations, units, and labels.
• Outliers: Detected using statistical measures, box plots, -scores, interquartile range, clustering, or distance-based methods. For the interquartile range method, observations outside are potential outliers.

An outlier should not be removed automatically. It may represent an error, a rare valid event, or an important discovery. Treatment can include correction, capping, transformation, exclusion, or separate analysis.

Data integration is the process of combining data from databases, files, applications, or external sources into a unified and consistent dataset.

Major integration issues include:

• Schema mismatch: Equivalent attributes may have different names, types, or structures.
• Entity identification: Different records may refer to the same real-world object.
• Value conflicts: Sources may use different units, formats, codes, or levels of precision.
• Redundancy: The same information may occur in several attributes or records.
• Semantic differences: Identically named attributes may have different meanings.
• Data-quality differences: Sources may vary in completeness, accuracy, and timeliness.

Integration requires schema matching, metadata analysis, entity resolution, unit conversion, conflict resolution, and redundancy removal. The goal is to produce a coherent dataset without losing relevant information.

Schema integration combines the structures of multiple data sources, whereas entity identification determines which records represent the same real-world object.

• Schema integration: Resolves differences in attribute names, data types, relationships, and table structures. For example, one database may use customer_id, while another uses client_number for the same concept.
• Entity identification: Matches records even when their values are not identical. For example, A. Sharma, 12 Park Rd. and Anita Sharma, 12 Park Road may represent the same customer.

Schema integration commonly uses metadata, mappings, and domain definitions. Entity identification uses keys, similarity measures, probabilistic matching, and domain-specific rules.

Both are necessary because matching attributes does not guarantee that corresponding records have been matched, and matching records is difficult without understanding their schemas.

Redundant data represents the same or derivable information more than once. It may arise from duplicate attributes, duplicate tuples, or attributes that can be calculated from others.

For numerical attributes and , Pearson's correlation coefficient is:

• A value near indicates strong positive correlation.
• A value near indicates strong negative correlation.
• A value near indicates weak linear correlation.

Highly correlated attributes may contain overlapping information, although correlation alone does not prove redundancy. Categorical attributes can be examined using chi-square tests or association measures. Metadata, functional dependencies, and domain knowledge should also be used before deleting an attribute.

Data-value conflicts occur when sources provide different representations or values for the same attribute or entity. They can be resolved as follows:

• Standardize formats: Convert dates, phone numbers, addresses, and codes to common formats.
• Convert units: Transform values such as pounds to kilograms or Fahrenheit to Celsius.
• Align scales: Ensure numerical values use the same base, precision, and measurement convention.
• Map categories: Convert labels such as M/F and Male/Female into a shared coding system.
• Apply source priorities: Prefer the most authoritative, accurate, or recent source.
• Use reconciliation rules: Select the latest value, majority value, average value, or domain-approved value.
• Preserve provenance: Record the source and transformation applied to each integrated value.

Conflicts should be resolved using explicit, repeatable rules rather than arbitrary choices. Unresolved conflicts should be flagged for expert review.

Data transformation converts data into forms that are more suitable for storage, mining, modeling, or interpretation.

Major forms include:

• Smoothing: Removes noise from data.
• Aggregation: Summarizes detailed records, such as converting daily sales into monthly sales.
• Generalization: Replaces low-level values with higher-level concepts, such as city with state.
• Normalization: Rescales numerical attributes to a common range or distribution.
• Attribute construction: Creates new features from existing ones, such as total_price = quantity × unit_price.
• Encoding: Converts categorical values into numerical representations.
• Mathematical transformation: Uses logarithmic, square-root, or power transformations to reduce skewness or stabilize variance.

Transformation can improve model performance, comparability, computational efficiency, and pattern discovery.

The three methods rescale numerical attributes in different ways.

1. Min-max normalization maps a value from to :

It preserves relative spacing but is sensitive to outliers and future values outside the original range.

2. Z-score normalization uses the mean and standard deviation :

It produces a mean of approximately and a standard deviation of . It is useful when the range is unknown.

3. Decimal scaling moves the decimal point:

Here, is the smallest integer such that the maximum absolute normalized value is less than .

Min-max is useful for fixed-range inputs, z-score for many statistical and machine-learning methods, and decimal scaling for simple order-of-magnitude adjustment.

These operations transform data at different levels:

• Aggregation: Combines multiple detailed values into a summary. For example, hourly electricity readings can be aggregated into daily consumption totals. It reduces volume but may hide local variation.
• Generalization: Replaces specific values with higher-level concepts using a hierarchy. For example, Chennai may become Tamil Nadu, then India.
• Attribute construction: Produces a new feature from existing attributes. For example, body mass index can be constructed as:

Aggregation mainly changes granularity, generalization changes abstraction level, and attribute construction changes the feature representation. All three can simplify patterns and improve analysis when applied with suitable domain knowledge.

Many analytical algorithms require numerical inputs, while raw variables may be categorical or strongly skewed.

Categorical encoding techniques:

• Label encoding: Assigns an integer to each category. It is compact but may introduce a false order.
• One-hot encoding: Creates one binary feature per category. It avoids false ordering but can produce many dimensions.
• Ordinal encoding: Assigns ordered values to naturally ranked categories.
• Frequency encoding: Replaces a category with its occurrence frequency, but different categories may receive the same value.

Nonlinear transformations:

• A logarithmic transformation such as reduces right skew and the influence of large values.
• Square-root and power transformations can stabilize variance.

Transformations must be fitted using training data and applied consistently to validation and test data to prevent data leakage.

Data reduction produces a smaller representation of a dataset while preserving the information needed for analysis. It reduces storage, computation, and model-training time.

Major strategies are:

• Dimensionality reduction: Reduces the number of attributes using feature selection or feature extraction.
• Numerosity reduction: Replaces data with smaller representations such as samples, clusters, histograms, or regression models.
• Data-cube aggregation: Stores summarized data at higher levels of a multidimensional cube.
• Compression: Encodes data using fewer bits through lossless or lossy methods.
• Discretization and concept hierarchies: Replace numerous continuous or detailed values with intervals or higher-level concepts.

A good reduction method balances efficiency with information preservation. Its quality can be evaluated by comparing accuracy, reconstruction error, query results, or statistical properties before and after reduction.

Principal Component Analysis transforms correlated numerical attributes into a smaller set of uncorrelated components that preserve maximum variance.

Steps:

1. Arrange the observations in a data matrix .
2. Center each feature by subtracting its mean; standardize features if their scales differ greatly.
3. Compute the covariance matrix:

4. Find eigenvalues and eigenvectors satisfying:

5. Sort the eigenvectors by decreasing eigenvalues.
6. Select the first eigenvectors to form the projection matrix .
7. Project the original data:

The explained-variance ratio retained by components is:

PCA reduces dimensions and multicollinearity, but components can be difficult to interpret and may not preserve nonlinear structure.

Numerosity reduction replaces a large dataset with a smaller collection of records or parameters.

• Sampling: Selects a representative subset. Simple random, stratified, and systematic sampling are common. It is efficient but may miss rare patterns.
• Histograms: Divide an attribute into buckets and store frequencies or summaries. They are compact but lose details within each bucket.
• Clustering: Represents similar objects through cluster centroids, medoids, and sizes. It preserves group structure but depends on the distance measure and clustering method.
• Regression: Represents data using a fitted function and its parameters. For linear regression, summarizes the relationship between two variables. It is effective when a suitable functional relationship exists.

Sampling retains actual records, while the other methods generally store summaries or models. The best method depends on the intended query, distribution, and acceptable information loss.

Data-cube aggregation summarizes multidimensional data at higher levels. For example, sales stored by day, product, and city may be aggregated by month, category, and state. Queries on summarized levels become faster because fewer cells are processed. However, detailed information may no longer be available from the aggregate alone.

Data compression represents information using fewer bits:

• Lossless compression: The original data can be reconstructed exactly. It is appropriate for identifiers, text, and records requiring complete accuracy.
• Lossy compression: Some detail is discarded to achieve a greater reduction. It is often used for images, signals, or approximate analytics.
• Transform-based compression: Data is transformed into another representation, and less important coefficients may be removed.

Cube aggregation reduces logical detail, whereas compression reduces physical representation size. Both reduce storage and processing costs, but only suitable methods should be used when exact recovery is required.

Data discretization converts a continuous numerical attribute into a finite set of intervals, with each value replaced by its interval label.

Objectives include:

• Reducing the number of distinct values.
• Simplifying models and improving interpretability.
• Reducing the influence of small measurement variations.
• Supporting algorithms that require categorical inputs.
• Creating concept hierarchies.

Unsupervised discretization ignores class labels. Equal-width and equal-frequency binning are examples. It is simple but may create intervals that are not useful for prediction.

Supervised discretization uses class labels to select boundaries that improve class separation. Entropy-based and chi-square-based methods are examples.

Discretization may also be top-down, starting with one interval and splitting it, or bottom-up, starting with many intervals and merging adjacent ones.

Equal-width discretization divides the complete range into intervals of identical width. For minimum , maximum , and bins, the width is:

One valid boundary convention gives:

• :
• :
• :
• :

Equal-frequency discretization places approximately the same number of observations in each bin. With eight values and four bins, each bin contains two values:

• Bin 1:
• Bin 2:
• Bin 3:
• Bin 4:

Equal-width bins are easy to interpret but can be unbalanced for skewed data. Equal-frequency bins balance record counts but may have unequal widths and may split identical values unless ties are handled carefully.

Entropy-based discretization is a supervised, top-down technique that selects boundaries producing pure class partitions.

For a dataset with classes, entropy is:

where is the proportion of records belonging to class .

For a candidate split point , divide into and . The expected entropy after splitting is:

The information gain is:

The split with the largest information gain, equivalently the smallest weighted post-split entropy, is selected. The process can be repeated recursively until a stopping condition is met, such as minimum gain, maximum number of intervals, or minimum interval size.

This method creates class-relevant intervals but may overfit unless splitting is controlled.

A concept hierarchy organizes attribute values into multiple levels of abstraction. It allows detailed data to be replaced with broader concepts.

Examples include:

• Location: street city state country.
• Time: day month quarter year.
• Age: exact age age interval life-stage category.

Hierarchies can be generated through:

• Domain knowledge: Experts define meaningful levels.
• Database schemas: Existing relationships between tables define levels.
• Cardinality analysis: Attributes with many distinct values are placed at lower levels, while attributes with fewer values are placed higher.
• Discretization: Continuous values are divided into intervals that serve as higher-level categories.

Concept hierarchies support data reduction, roll-up operations, multilevel mining, understandable reports, and generalized rules. Poorly chosen hierarchies can hide useful detail, so levels and boundaries should reflect the analytical objective.