Unit 2: Data Pre-processing - Subjective Questions

Data preprocessing is the process of inspecting, cleaning, transforming, and organizing raw data before it is used for analysis or model development.

Importance:

• Improves the accuracy and reliability of analysis.
• Identifies missing, duplicate, inconsistent, or invalid values.
• Converts data into formats suitable for algorithms.
• Reduces noise and irrelevant information.
• Prevents misleading conclusions caused by poor-quality data.
• Makes data collected from different sources consistent.

Since model quality depends heavily on input quality, data preprocessing is commonly summarized by the principle garbage in, garbage out.

A typical data preprocessing workflow contains the following steps:

1. Data collection: Obtain data from databases, files, APIs, sensors, or surveys.
2. Data profiling: Examine structure, data types, distributions, and quality issues.
3. Data cleaning: Handle missing values, duplicates, errors, noise, and outliers.
4. Data integration: Combine information from multiple sources.
5. Data transformation: Scale, normalize, encode, aggregate, or reshape variables.
6. Data reduction: Remove irrelevant attributes or reduce dimensionality.
7. Data validation: Confirm that constraints, formats, and relationships are correct.
8. Documentation: Record transformations so that the process is reproducible.

These steps may be repeated because preprocessing is generally an iterative process.

Data wrangling is the practical process of converting raw, complex, or poorly structured data into a clean and usable form.

Distinction:

• Data preprocessing is a broad concept covering cleaning, transformation, integration, reduction, and preparation for analysis.
• Data wrangling emphasizes acquiring, restructuring, merging, filtering, and transforming data into a convenient shape.
• Wrangling often deals with operational tasks such as splitting columns, reshaping tables, parsing dates, and joining datasets.
• Preprocessing may additionally include machine-learning-specific operations such as scaling, feature selection, and encoding.

Thus, data wrangling can be considered an important subprocess within the wider preprocessing stage.

Structured data:

• Follows a fixed schema of rows and columns.
• Is easy to query using database tools.
• Examples include relational tables, spreadsheets, and transaction records.

Semi-structured data:

• Does not follow a rigid tabular schema but contains tags, keys, or metadata.
• Its records may have different attributes.
• Examples include JSON, XML, emails, and system logs.

Unstructured data:

• Has no predefined organizational model.
• Requires specialized processing to extract useful features.
• Examples include text documents, images, audio, and videos.

The required preprocessing depends on the form: structured data may need column cleaning, semi-structured data requires parsing, and unstructured data usually requires feature extraction.

Data is broadly classified as categorical or numerical.

Categorical data:

• Represents labels, classes, or qualities.
• Nominal: Categories have no natural order, such as color or city.
• Ordinal: Categories have a meaningful order, such as low, medium, and high.

Numerical data:

• Represents measurable quantities on which arithmetic may be performed.
• Discrete: Contains countable values, such as number of students.
• Continuous: Can take any value within a range, such as height or temperature.

Correct identification matters because it determines suitable preprocessing. Categorical data may require encoding, while numerical data may require scaling, missing-value treatment, or outlier detection.

The four common scales of measurement are:

• Nominal: Values identify categories without order. Examples are blood group and country. Only equality comparisons are meaningful.
• Ordinal: Values identify ordered categories, but differences between levels are not necessarily equal. Examples are satisfaction ratings and class ranks.
• Interval: Values have ordered, equal intervals but no true zero. Temperature in Celsius is an example; differences are meaningful, but ratios are not.
• Ratio: Values have equal intervals and a true zero. Examples include weight, distance, age, and income. Both differences and ratios are meaningful.

Understanding these scales helps select valid statistical operations, visualizations, and transformations.

Common data error types include:

• Missing-value errors: A customer's age is absent.
• Typographical errors: A city is entered as Dehli instead of Delhi.
• Format errors: Dates appear as both 10-09-2026 and 2026/09/10.
• Range errors: A percentage is recorded as 140.
• Type errors: Text such as unknown appears in a numeric column.
• Duplicate errors: The same transaction is stored more than once.
• Consistency errors: The same category is represented by M, Male, and male.
• Referential errors: An order refers to a customer ID that does not exist.
• Measurement errors: A faulty sensor produces inaccurate readings.
• Outliers and noise: Values are unusually extreme or randomly distorted.

Identifying the error category helps determine the appropriate correction method.

Missing data can be handled using several methods:

• Delete rows: Suitable when only a small, random proportion of records is missing.
• Delete columns: Appropriate when an attribute has excessive missingness and little analytical value.
• Mean or median imputation: Used for numerical data; the median is more resistant to outliers.
• Mode imputation: Suitable for categorical variables.
• Group-based imputation: Replaces values using statistics calculated within relevant groups.
• Forward or backward filling: Useful for ordered time-series data when nearby observations are related.
• Interpolation: Estimates values between known time points.
• Model-based imputation: Predicts missing values using other variables.
• Missing indicator: Adds a variable showing whether the original value was missing.

The choice must consider the amount and cause of missingness because careless imputation can introduce bias.

Duplicate records are repeated observations representing the same real-world entity or event.

Detection methods:

• Check for exact equality across all columns.
• Search for repeated primary keys or transaction IDs.
• Compare selected identifying fields such as name, date, and contact number.
• Use approximate or fuzzy matching for spelling and formatting variations.
• Apply entity-resolution rules to records from different sources.

Treatment methods:

• Remove exact duplicates when they have no independent meaning.
• Retain one authoritative or most recent record.
• Merge complementary fields from partial duplicates.
• Investigate repeated events before deletion because identical values do not always mean duplicate events.
• Document the deduplication rule for reproducibility.

Unresolved duplicates can inflate counts, distort distributions, and bias models.

Inconsistencies arise when data is entered using different conventions, formats, units, spellings, or coding schemes. For example, gender may be recorded as Female, F, and female, while weight may appear in both kilograms and pounds.

Standardization methods:

• Convert text to a consistent letter case.
• Remove unwanted spaces and characters.
• Map synonymous category labels to one canonical value.
• Convert dates to a common format such as ISO YYYY-MM-DD.
• Convert measurements to common units.
• Apply controlled vocabularies and lookup tables.
• Enforce schema, range, and domain constraints.
• Reconcile conflicting values using an authoritative source.

Standardization improves comparison, grouping, joining, and statistical analysis.

An outlier is an observation that differs substantially from most other observations. It may represent a genuine rare event or an error.

Detection methods:

• Visual inspection using box plots, scatter plots, or histograms.
• Domain-based minimum and maximum limits.
• The interquartile range rule, where values below or above are flagged.
• Standardized scores, where unusually large absolute -scores are examined.
• Multivariate methods that consider unusual combinations of variables.

Treatment:

• Correct the value if it is a recording error.
• Remove it only with clear justification.
• Cap extreme values at selected limits.
• Apply transformations such as logarithms.
• Use robust statistics or models.
• Retain genuine unusual observations when they are relevant.

Outliers should be investigated rather than automatically deleted.

Data noise refers to random variation or irrelevant fluctuations that obscure the underlying pattern. A data error is an incorrect value caused by faults such as invalid entry, coding mistakes, or equipment failure.

For example, small random variations in repeated sensor readings may be noise, while a negative value from a broken sensor may be an error.

Noise-treatment methods:

• Smooth numerical data using moving averages or binning.
• Apply signal filters when processing sensor or time-series data.
• Aggregate observations over suitable intervals.
• Use robust statistical measures such as the median.
• Remove irrelevant attributes or features.
• Correct systematic measurement problems through calibration.
• Flag suspicious observations for manual review.

Excessive smoothing should be avoided because it can remove genuine patterns and rare events.

Validation rules describe the conditions that valid data must satisfy.

Common rules include:

• Type constraints: Age must be numeric.
• Range constraints: Percentage must satisfy .
• Domain constraints: Status must belong to an approved set of categories.
• Format constraints: An email or date must follow an accepted pattern.
• Uniqueness constraints: A primary key must not repeat.
• Referential constraints: A foreign key must match an existing parent record.
• Cross-field constraints: A delivery date cannot be earlier than an order date.
• Completeness constraints: Mandatory fields cannot be empty.

Automated validation reports invalid records consistently and early. However, domain experts should define the rules because a statistically unusual value may still be valid.

Normalization commonly rescales a value to a fixed range, usually :

It preserves relative ordering and is useful when a bounded scale is desired. It is sensitive to extreme minimum and maximum values.

Standardization transforms data to have approximately zero mean and unit standard deviation:

It does not restrict values to a fixed interval and is useful for methods affected by feature scale, such as distance-based algorithms and many linear models.

Both methods prevent variables with large numerical magnitudes from dominating smaller-scale variables. Transformation parameters should be learned from the training data and then applied unchanged to validation and test data.

Many analytical and machine-learning algorithms require numeric inputs, so categorical values must be converted into numerical representations.

Common methods:

• Label encoding: Assigns an integer to each category. It is compact but may create a false order for nominal data.
• Ordinal encoding: Assigns ordered numbers to genuinely ranked categories.
• One-hot encoding: Creates one binary column per category. It avoids false ordering but can produce many columns.
• Frequency encoding: Replaces each category with its frequency. It is compact but may map distinct categories to the same value.
• Target encoding: Uses a target-based statistic. It can be effective but may cause target leakage and overfitting.

The method should reflect the variable's meaning, cardinality, model requirements, and risk of leakage. Unknown categories must also be handled consistently.

Data integration combines datasets into a unified representation. Major challenges include:

• Schema mismatch: Equivalent fields have different names or structures.
• Type mismatch: An identifier is numeric in one source and textual in another.
• Unit mismatch: Measurements use different units or scales.
• Format mismatch: Dates, addresses, and categories follow different conventions.
• Entity resolution: The same person or object lacks a shared identifier.
• Redundancy: Sources contain duplicate records or correlated attributes.
• Conflicting values: Sources provide different values for the same fact.
• Granularity mismatch: One dataset is daily while another is monthly.
• Referential problems: Keys are missing or do not correspond.

These issues are addressed through schema mapping, type conversion, unit standardization, matching rules, deduplication, aggregation, and authoritative-source policies.

These operations convert raw data into a form suitable for analysis:

• Filtering: Selects records satisfying conditions, such as transactions from a particular year.
• Sorting: Arranges records by one or more attributes to reveal sequence or ranking.
• Grouping: Divides data into categories so summary statistics can be calculated.
• Joining: Combines related tables using common keys, such as joining orders with customers.
• Reshaping: Converts data between wide and long forms or pivots categories into columns.

Together, these operations help create analytical tables, summarize observations, align entities, and prepare data for visualization or modeling. Their correctness depends on suitable keys, aggregation rules, and checks that rows have not been unintentionally lost or multiplied.

A suitable strategy is:

1. Preserve raw data: Store an unchanged copy and create a working version.
2. Profile the dataset: Inspect data types, missing percentages, unique values, distributions, and key frequencies.
3. Validate identifiers: Investigate duplicate customer IDs and determine whether records should be removed, merged, or retained as separate events.
4. Handle age: Check valid age ranges and analyze why values are missing. Use deletion or justified imputation, possibly within relevant groups.
5. Standardize cities: Trim spaces, normalize case, correct known spellings, and map aliases to canonical city names.
6. Parse dates: Convert all recognized formats to a standard date type and flag impossible or ambiguous dates.
7. Examine income: Verify units and data-entry errors; detect outliers using domain limits, plots, and robust statistical rules.
8. Transform if required: Scale or log-transform income for suitable models without destroying genuine high-income observations.
9. Validate results: Recheck uniqueness, completeness, ranges, formats, and row counts.
10. Document the process: Record every rule, threshold, and affected record for reproducibility.

The final decisions should combine statistical evidence with domain knowledge.

Data profiling is the systematic examination of a dataset's structure, content, distributions, and relationships before detailed cleaning or analysis.

A profile commonly includes:

• Row and column counts.
• Inferred and declared data types.
• Missing-value counts and percentages.
• Minimum, maximum, mean, median, and quantiles.
• Frequency tables and numbers of unique values.
• Duplicate-key counts.
• Pattern and format analysis.
• Distribution and outlier summaries.
• Relationships and correlations between fields.

Profiling can reveal unexpected nulls, impossible ranges, mixed data types, rare categories, formatting variations, duplicate identifiers, and suspicious distributions. It provides evidence for defining cleaning rules and creates a baseline against which the processed dataset can be validated.

A reliable solution requires preprocessing that is both correct and reproducible.

• Quality: Valid cleaning rules reduce errors and bias, while inappropriate deletion or imputation can distort results.
• Documentation: Each source, assumption, transformation, threshold, and exception should be recorded.
• Reproducibility: Preprocessing should be implemented as repeatable code or pipelines rather than undocumented manual edits.
• Traceability: Processed values should be traceable to raw records and transformation rules.
• Versioning: Data, code, schemas, and configuration should have identifiable versions.
• Testing: Pipelines should check ranges, types, uniqueness, row counts, and expected outputs.
• Leakage prevention: Transformations must not use unavailable future information or test-set statistics.

These practices support auditing, collaboration, consistent model deployment, and detection of data drift when new data arrives.