Unit 1: Data preprocessing and visualization - Subjective Questions

Data refers to raw facts, figures, or observations collected from various sources that can be processed to extract meaningful information.

Types of Data:

1. Qualitative (Categorical) Data – Describes qualities or categories.

   • Nominal Data: Categories with no natural order. Example: gender (Male, Female), colors (Red, Blue).
   • Ordinal Data: Categories with a meaningful order but no fixed interval. Example: ratings (Poor, Average, Good), education level.

2. Quantitative (Numerical) Data – Represents measurable quantities.

   • Discrete Data: Countable values. Example: number of students, cars in a lot.
   • Continuous Data: Any value within a range. Example: height, weight, temperature.

Additional classification:

• Structured Data: Organized in rows and columns (e.g., databases).
• Unstructured Data: No predefined format (e.g., images, text, audio).

Understanding data types is crucial because it dictates the choice of statistical methods, visualizations, and preprocessing techniques.

Data preprocessing is the process of transforming raw data into a clean, usable, and meaningful format before analysis or modeling.

Key steps in preprocessing:

• Data Cleaning: Handling missing values, removing noise, and correcting inconsistencies.
• Data Integration: Combining data from multiple sources.
• Data Transformation: Normalization, scaling, and encoding.
• Data Reduction: Reducing dimensionality and volume while preserving information.

Why it is critical:

• Garbage In, Garbage Out (GIGO): Poor quality data leads to inaccurate models.
• Improves Accuracy: Clean data enhances model performance and reliability.
• Handles Real-World Imperfections: Real data is often incomplete, noisy, and inconsistent.
• Efficiency: Reduces computation time by eliminating redundant features.

Studies suggest data scientists spend nearly 60–80% of their time on data preprocessing, highlighting its importance.

Missing data occurs when no value is stored for a variable in an observation.

Common causes:

• Errors during data entry.
• Equipment or sensor failures.
• Respondents skipping survey questions.
• Data corruption or loss during transfer.
• Merging incompatible datasets.

Mechanisms of Missingness:

1. MCAR (Missing Completely At Random): The probability of missing data is unrelated to any variable. Example: a sample tube accidentally breaks.

2. MAR (Missing At Random): Missingness depends on observed data but not on the missing value itself. Example: men are less likely to report weight, but missingness depends on gender (observed).

3. MNAR (Missing Not At Random): Missingness depends on the unobserved value itself. Example: high-income earners refusing to disclose income.

Identifying the mechanism helps in selecting the appropriate imputation strategy.

Techniques for handling missing data:

1. Deletion Methods:

• Listwise Deletion: Remove entire rows with any missing value. Simple but causes data loss.
• Pairwise Deletion: Use available data for each analysis without discarding entire rows.

2. Imputation Methods:

• Mean/Median/Mode Imputation: Replace missing values with central tendency measures.
• Forward/Backward Fill: Used in time-series data.
• Regression Imputation: Predict missing values using regression models.
• KNN Imputation: Use nearest neighbors to estimate values.
• Multiple Imputation: Generate multiple plausible datasets and combine results.

Comparison:

Conclusion: Deletion is easy but wasteful; imputation preserves data but may introduce estimation error if done improperly.

A scatter plot is a two-dimensional data visualization that uses Cartesian coordinates to display values of two numerical variables. Each point represents an observation with its and values.

Purpose:

• To examine the relationship between two continuous variables.
• To detect patterns, trends, clusters, and outliers.

Identifying Correlation:

• Positive Correlation: Points slope upward from left to right; as increases, increases.
• Negative Correlation: Points slope downward; as increases, decreases.
• No Correlation: Points are scattered randomly with no discernible pattern.

The strength can be quantified by the correlation coefficient , where :

Example: A scatter plot of hours studied vs exam score typically shows a positive correlation.

A histogram is a graphical representation of the frequency distribution of continuous numerical data, where data is grouped into bins (intervals).

Construction:

1. Divide the data range into equal intervals (bins).
2. Count the number of observations in each bin (frequency).
3. Draw adjacent bars whose heights represent the frequency.

Interpretation:

• Reveals the shape of the distribution (normal, skewed, bimodal).
• Shows central tendency and spread.
• Helps detect outliers and gaps.

Histogram vs Bar Chart:

Note: The number of bins greatly affects interpretation; too few oversimplifies, too many adds noise.

Both histograms and scatter plots are fundamental visualization tools but serve different purposes.

Histogram:

• Represents the distribution of a single variable.
• Uses bins to group continuous data.
• Answers: How is the data distributed? Is it skewed?
• Shows frequency, central tendency, and spread.

Scatter Plot:

• Represents the relationship between two variables.
• Uses individual points for each observation.
• Answers: Are the two variables related? Is there correlation?
• Shows trends, clusters, and outliers in bivariate data.

Comparison Table:

Summary: Use a histogram to understand one variable's spread and a scatter plot to explore relationships between two variables.

Group plots (also called grouped plots or faceted plots) are visualizations that display data separated by one or more categorical variables, enabling side-by-side comparison.

Types of Group Plots:

• Grouped Bar Charts: Bars for different subgroups placed side by side.
• Faceting (Small Multiples): A grid of small charts, each for a subset of data.
• Grouped Box Plots: Multiple box plots for different categories.
• Grouped Scatter Plots: Points colored/shaped by category.

Benefits:

• Comparative Analysis: Easily compare distributions or trends across categories.
• Pattern Detection: Reveals differences between subgroups (e.g., sales by region).
• Reduces Clutter: Faceting separates overlapping data into distinct panels.
• Multivariate Insight: Incorporates categorical dimensions alongside numerical ones.

Example: Comparing exam scores across different classes using grouped box plots highlights which class performs better and its variability.

Best Practice: Keep consistent scales across facets for fair comparison.

A box plot is a standardized visualization that displays the distribution of numerical data using five-number summary statistics.

Five-Number Summary:

1. Minimum – smallest non-outlier value.
2. First Quartile () – 25th percentile.
3. Median () – 50th percentile.
4. Third Quartile () – 75th percentile.
5. Maximum – largest non-outlier value.

Components:

• Box: Spans from to , representing the Interquartile Range (IQR).
• Median Line: A line inside the box.
• Whiskers: Extend to the minimum and maximum within .
• Outliers: Points beyond the whiskers, plotted individually.

Outlier Detection:

• Lower Bound:
• Upper Bound:

Any value outside these bounds is considered an outlier.

Use: Box plots are excellent for comparing distributions and spotting skewness and outliers.

Given dataset (sorted): ()

Step 1: Median ()
With even , median is the average of the 3rd and 4th values:

Step 2: First Quartile ()
Lower half: ; median = .

Step 3: Third Quartile ()
Upper half: ; median = .

Step 4: Five-Number Summary

• Minimum =
• Median =
• Maximum =

Step 5: IQR

Step 6: Outlier Bounds

• Lower:
• Upper:

Conclusion: All values lie within , so there are no outliers.

Dimensionality reduction is the process of reducing the number of input variables (features) in a dataset while retaining as much meaningful information as possible.

Need for Dimensionality Reduction:

• Reduce Computation: Fewer features mean faster training.
• Avoid Overfitting: Fewer irrelevant features improve generalization.
• Visualization: Reduce to 2D/3D for plotting.
• Remove Redundancy: Eliminate correlated/noisy features.

Curse of Dimensionality:
As the number of dimensions increases:

• Data becomes sparse; points appear equidistant.
• Distance metrics lose meaning, harming algorithms like KNN.
• Volume grows exponentially, requiring exponentially more data.
• Overfitting risk increases dramatically.

Two Approaches:

1. Feature Selection: Choose a subset of original features.
2. Feature Extraction: Transform features into a lower-dimensional space (e.g., PCA).

Conclusion: Dimensionality reduction improves efficiency, accuracy, and interpretability of models.

Principal Component Analysis (PCA) is a linear feature extraction technique that transforms correlated variables into a set of uncorrelated variables called principal components, ordered by the variance they capture.

Key Idea: Project data onto directions (principal components) that maximize variance.

Main Steps:

1. Standardize the Data: Center and scale features to zero mean and unit variance.

2. Compute Covariance Matrix:
   

3. Eigen Decomposition: Calculate eigenvalues () and eigenvectors () of the covariance matrix:
   

4. Sort Eigenvectors: Rank by eigenvalues in descending order. Larger eigenvalues capture more variance.

5. Select Top Components: Choose the top eigenvectors to form the projection matrix.

6. Transform Data: Project original data onto the new subspace.

Advantages: Reduces dimensions, removes correlation, and aids visualization.
Limitation: It is linear and components may be hard to interpret.

Both are approaches to dimensionality reduction but differ fundamentally in how they reduce features.

Feature Selection:

• Selects a subset of the original features.
• Original features are preserved (interpretable).
• Methods:
  • Filter Methods: Correlation, Chi-square, Information Gain.
  • Wrapper Methods: Recursive Feature Elimination (RFE).
  • Embedded Methods: LASSO regularization.

Feature Extraction:

• Transforms features into a new, lower-dimensional space.
• Creates new features (combinations of originals) that may lose interpretability.
• Methods: PCA, LDA, t-SNE, Autoencoders.

Comparison Table:

Conclusion: Use feature selection when interpretability matters and feature extraction when maximizing information compression.

The four scales of measurement classify data based on their properties.

1. Nominal Scale:

• Categories with no order.
• Only equality/labels apply.
• Example: Gender, Eye color, Blood type.

2. Ordinal Scale:

• Ordered categories, but intervals are not equal.
• Example: Rankings (1st, 2nd), Satisfaction (Low, Medium, High).

3. Interval Scale:

• Ordered with equal intervals, but no true zero.
• Example: Temperature in Celsius, Calendar dates. ( does not mean 'no temperature').

4. Ratio Scale:

• Ordered, equal intervals, and a true zero.
• Ratios are meaningful.
• Example: Height, Weight, Age, Income.

Comparison Table:

Understanding scales guides the selection of appropriate statistical operations.

Outliers are data points that significantly deviate from other observations.

Impact on Analysis:

• Distort Statistics: Mean and standard deviation get heavily skewed.
• Mislead Models: Regression lines get pulled toward outliers.
• Reduce Accuracy: Machine learning models may learn noise.
• Affect Visualizations: Compress scales and hide patterns.

Detection Methods:

1. Statistical (IQR Method): Values outside or .
2. Z-Score Method: Values with where .
3. Visualization: Box plots and scatter plots.
4. Clustering: Points far from clusters (e.g., DBSCAN).

Handling Methods:

• Removal: Delete if they are errors.
• Transformation: Apply log/square-root to reduce skew.
• Capping (Winsorization): Replace extreme values with boundary values.
• Imputation: Treat as missing and impute.

Note: Not all outliers are errors; some represent important rare events (e.g., fraud).

Normalization and standardization are feature scaling techniques used during data preprocessing to bring features onto a comparable scale.

Normalization (Min-Max Scaling):

• Rescales data to a fixed range, typically .
  
• Sensitive to outliers.

Standardization (Z-score Scaling):

• Transforms data to have mean and standard deviation .
  
• Less sensitive to outliers; does not bound values.

Why Important:

• Prevents features with large ranges from dominating.
• Improves convergence of gradient-based algorithms.
• Essential for distance-based algorithms (KNN, SVM, K-Means).

Differences:

Conclusion: Choose based on the algorithm and the data distribution.

Box plots are powerful for comparing the distribution of a numerical variable across multiple groups side by side.

Interpreting Skewness:

• Symmetric: Median is centered in the box; whiskers are equal length.
• Right (Positive) Skew: Median closer to ; longer upper whisker.
• Left (Negative) Skew: Median closer to ; longer lower whisker.

Interpreting Spread:

• Box Length (IQR): Larger box = greater variability in the middle 50%.
• Whisker Length: Indicates the range of non-outlier data.
• Outliers: Individual points reveal extreme values.

Comparing Groups:

• Median Position: Compare central tendencies across groups.
• Box Sizes: Compare variability; wider boxes = more spread.
• Overlap: Little overlap suggests groups differ significantly.

Example Interpretation Guide:

Use Case: Comparing salaries across departments quickly reveals which department pays more and has higher inequality.

Data visualization is central to Exploratory Data Analysis (EDA), the process of summarizing and understanding data before formal modeling.

Role of Visualization in EDA:

• Pattern Recognition: Reveals trends, cycles, and clusters.
• Anomaly Detection: Spots outliers and errors.
• Distribution Understanding: Shows shape, skew, and modality.
• Relationship Discovery: Uncovers correlations between variables.
• Communication: Presents findings intuitively to stakeholders.

Key Plots:

• Histogram: Shows distribution of a single continuous variable.
• Scatter Plot: Reveals relationships between two numerical variables.
• Box Plot: Summarizes spread, median, and outliers.
• Bar Chart: Compares categorical frequencies.
• Group/Faceted Plots: Compare distributions across categories.
• Heatmap: Visualizes correlation matrices.
• Line Plot: Displays trends over time.

Conclusion: Visualization transforms raw numbers into actionable insights and guides subsequent preprocessing and modeling decisions.

Both PCA and t-SNE reduce dimensionality but operate on fundamentally different principles.

PCA (Principal Component Analysis):

• Type: Linear technique.
• Objective: Maximize variance along orthogonal components.
• Strengths: Fast, deterministic, preserves global structure, good for preprocessing.
• Weaknesses: Cannot capture non-linear relationships.

t-SNE (t-Distributed Stochastic Neighbor Embedding):

• Type: Non-linear technique.
• Objective: Preserve local neighborhood structure by matching probability distributions.
• Strengths: Excellent for visualizing clusters in 2D/3D.
• Weaknesses: Computationally expensive, non-deterministic, poor at preserving global distances, sensitive to hyperparameters (perplexity).

Comparison Table:

Conclusion: Use PCA for feature reduction and speed; use t-SNE for visualizing complex, high-dimensional clusters.

Choosing an imputation strategy depends on the data type, distribution, and presence of outliers.

1. Mean Imputation:

• Replaces missing values with the arithmetic mean.
• Use When: Data is numerical and symmetrically distributed without outliers.
• Drawbacks: Highly sensitive to outliers; reduces variance; distorts distribution.

2. Median Imputation:

• Replaces missing values with the median (middle value).
• Use When: Data is numerical and skewed or contains outliers.
• Drawbacks: Ignores relationships between variables; still reduces variability.

3. Mode Imputation:

• Replaces missing values with the most frequent value.
• Use When: Data is categorical or discrete.
• Drawbacks: Can over-represent the dominant category; unsuitable for continuous data.

Decision Logic:

• If categorical → use mode.
• If numerical & symmetric → use mean.
• If numerical & skewed / outliers present → use median.

General Drawback: All single-value imputation methods underestimate variance and ignore feature correlations. Advanced methods like KNN or Multiple Imputation address this better.