Unit 2: Data Preparation and Machine Learning Workflow - Subjective Questions

NumPy is a Python library used for numerical computing and forms the foundation of many machine learning tools.

Important features:

• Provides multidimensional arrays called ndarray.
• Performs fast mathematical and statistical operations.
• Supports vectorization, which reduces the need for explicit loops.
• Provides functions for reshaping, indexing, sorting, and aggregating data.
• Supports linear algebra, random number generation, and matrix operations.

Common operations:

• Creating arrays using functions such as array(), zeros(), and ones().
• Reshaping data using reshape().
• Calculating means, sums, minimums, and maximums.
• Combining arrays using concatenation.
• Generating random data for simulations and model testing.

NumPy is useful because machine learning datasets are usually represented as numerical matrices, and NumPy provides efficient operations on these structures.

Pandas is a Python library used for loading, cleaning, transforming, analyzing, and organizing structured data.

• A Series is a one-dimensional labeled data structure, similar to a single column.
• A DataFrame is a two-dimensional labeled data structure consisting of rows and columns.

Uses of Pandas include:

• Reading data from CSV, Excel, JSON, and database files.
• Inspecting data using functions such as head(), info(), and describe().
• Selecting rows and columns using labels or conditions.
• Detecting and handling missing values.
• Sorting, grouping, merging, and aggregating data.
• Converting data types and removing duplicate records.

Pandas makes data preparation easier because it combines numerical processing with meaningful row and column labels.

Matplotlib is a Python visualization library used to represent data graphically. It helps analysts identify trends, distributions, relationships, and unusual observations.

Common plots include:

• Line plot: Shows trends or changes over an ordered variable such as time.
• Bar chart: Compares values across discrete categories.
• Histogram: Displays the frequency distribution of a numerical variable.
• Scatter plot: Shows the relationship between two numerical variables and can reveal correlation or clusters.
• Box plot: Summarizes the median, quartiles, spread, and possible outliers.
• Pie chart: Represents proportions of categories, although it should be used carefully.

Labels, titles, legends, and appropriate scales should be added to make graphs understandable. Visualization is an important part of EDA because patterns may not be obvious from raw tables.

Datasets can be classified according to how their information is organized.

• Structured datasets: Data is arranged in rows and columns with a fixed schema. Examples include a student marks table, a bank transaction table, or a relational database.
• Semi-structured datasets: Data does not follow a strict tabular structure but contains tags, keys, or metadata. Examples include JSON files, XML documents, and web API responses.
• Unstructured datasets: Data has no predefined tabular organization. Examples include images, audio recordings, videos, emails, and free-form text.

Datasets may also be classified as:

• Numerical or categorical, based on the feature values.
• Labeled or unlabeled, based on whether target values are available.
• Training, validation, or testing datasets, based on their role in model development.

The dataset type determines the tools, preprocessing methods, and algorithms that can be applied.

Data collection is the process of obtaining relevant observations for solving a machine learning problem.

Main steps:

1. Define the problem and identify the required target variable.
2. Determine the population and sample from which data will be collected.
3. Select appropriate sources such as databases, sensors, surveys, APIs, or public repositories.
4. Establish a consistent data format and collection procedure.
5. Store the data securely with suitable documentation.
6. Check and improve the quality of the collected data.

Important quality factors:

• Accuracy: Values should represent the real-world situation.
• Completeness: Important records and attributes should not be missing.
• Consistency: Values should follow the same definitions and formats.
• Relevance: Features should relate to the prediction problem.
• Timeliness: Data should be sufficiently current.
• Representativeness: The sample should reflect the target population.
• Ethical and legal compliance: Privacy, consent, fairness, and data protection must be considered.

Poorly collected data can produce biased models even when advanced algorithms are used.

Exploratory Data Analysis (EDA) is the process of examining and summarizing data before model training. Its purpose is to understand the dataset, identify quality problems, and discover useful patterns.

A systematic EDA procedure is:

• Inspect the number of rows and columns.
• Examine data types, column names, and unique values.
• Identify missing values and duplicate records.
• Calculate descriptive statistics such as mean, median, standard deviation, minimum, and maximum.
• Plot numerical distributions using histograms or density plots.
• Examine categorical frequencies using bar charts.
• Analyze relationships using scatter plots, grouped summaries, and correlation matrices.
• Detect possible outliers and data-entry errors.
• Check class balance when the target is categorical.
• Document findings and decide which preprocessing steps are required.

EDA should be performed carefully to avoid using information from the test set when making modeling decisions.

Missing values occur when an attribute has no recorded value. The appropriate treatment depends on the amount, pattern, and meaning of the missingness.

• Deletion: Rows or columns containing missing values are removed. It is simple but may waste data and introduce bias if missingness is systematic.
• Mean imputation: Missing numerical values are replaced with the mean. It is easy to use but can reduce variation and is sensitive to outliers.
• Median imputation: Values are replaced with the median. It is more robust for skewed distributions and outliers.
• Mode imputation: Missing categorical values are replaced with the most frequent category. It may overrepresent the majority class.
• Constant-value imputation: A value such as Unknown or zero is used when it has a meaningful interpretation.
• Model-based imputation: A regression model, nearest-neighbor method, or other predictive technique estimates missing values. It may preserve relationships but is more complex.

The imputation rule must be learned from the training data only and then applied to validation and test data to prevent data leakage.

An outlier is an observation that is unusually far from the general pattern of the data. It may result from measurement error, data-entry mistakes, unusual but valid behavior, or a rare event.

Detection methods:

• Interquartile Range method:
  • Compute the first quartile and third quartile .
  • Calculate .
  • Values below or above are potential outliers.
• Z-score method:
  • Compute the standardized value .
  • Values with an absolute z-score commonly greater than 3 may be considered unusual.

Treatment options:

• Correct obvious data-entry or measurement errors.
• Remove observations only when they are invalid and removal is justified.
• Cap or winsorize extreme values.
• Apply transformations such as logarithms.
• Use robust scaling or algorithms less sensitive to outliers.
• Keep valid extreme observations when they represent important real-world cases.

Outlier treatment should be based on domain knowledge rather than an automatic rule.

Feature scaling transforms numerical variables to comparable ranges. It is important when features have different units or magnitudes.

Why scaling is useful:

• Distance-based algorithms such as k-nearest neighbors and k-means can be dominated by large-valued features.
• Gradient-based algorithms often converge faster when variables are on similar scales.
• Regularized linear models treat coefficients more fairly when features are scaled.
• Principal component analysis is affected by feature magnitude.

Standardization:

It produces features with approximately mean zero and standard deviation one. It is often suitable when the data is not bounded and may contain different ranges.

Min-max normalization:

It generally maps values to the interval . It is useful when a bounded range is required but is sensitive to extreme values.

Scaling parameters must be calculated using training data only and reused for validation and test data.

Feature encoding converts categorical values into numerical representations that machine learning algorithms can process.

• Label encoding: Each category is assigned an integer. For example, Red, Blue, and Green may become 0, 1, and 2. It is appropriate for a target label or categories with a genuine order, but it can create a false numerical relationship for nominal categories.
• Ordinal encoding: Ordered categories are mapped according to their rank. For example, Low, Medium, and High may become 1, 2, and 3. This preserves meaningful order.
• One-hot encoding: A separate binary column is created for each category. For a color feature, columns such as Color_Red, Color_Blue, and Color_Green are created. Each row contains 0 or 1 values.

One-hot encoding is usually preferred for nominal categories because it does not impose an artificial order. High-cardinality categorical features may require alternatives such as frequency encoding or carefully applied target encoding.

Feature selection is the process of choosing a useful subset of the original features. It can improve model interpretability, reduce training time, and limit overfitting.

Feature selection methods:

• Filter methods: Select features using statistical properties without training a specific model. Examples include correlation analysis, chi-square tests, mutual information, and variance thresholds. They are fast and model-independent.
• Wrapper methods: Evaluate subsets of features by training a model repeatedly. Examples include forward selection, backward elimination, and recursive feature elimination. They can be accurate but computationally expensive.
• Embedded methods: Perform selection during model training. Examples include Lasso regularization, which can reduce some coefficients to zero, and tree-based feature importance.

Feature extraction creates new features from the original ones. Principal component analysis is an example because it transforms several variables into a smaller set of components. Feature selection keeps original variables, while feature extraction creates transformed variables.

A machine learning workflow is a sequence of activities used to develop and evaluate a reliable model.

1. Define the problem: Specify the objective, input features, target, constraints, and success metric.
2. Collect data: Obtain relevant and representative observations.
3. Understand the data: Perform EDA, inspect distributions, and identify quality issues.
4. Prepare the data: Handle missing values, outliers, duplicate records, inconsistent formats, and categorical variables.
5. Engineer and select features: Scale numerical variables, encode categories, and retain useful predictors.
6. Split the data: Create training and testing portions, with a validation strategy when needed.
7. Train models: Fit one or more suitable algorithms using the training data.
8. Validate and tune: Compare models and hyperparameters using validation or cross-validation.
9. Evaluate: Measure performance on the untouched test set.
10. Deploy and monitor: Use the model in practice and monitor drift, accuracy, fairness, and failures.

All preprocessing operations that learn parameters should be fitted within the training process to prevent data leakage.

A train-test split divides a dataset into two disjoint subsets:

• The training set is used to learn model parameters.
• The test set is kept unseen until final evaluation.

The split estimates how well the trained model may perform on new data. Common proportions include 80:20 or 70:30, although the choice depends on dataset size. For very large datasets, a smaller test proportion may be sufficient; for small datasets, cross-validation can provide a more stable estimate.

Stratification ensures that the class proportions in the training and test sets are approximately similar. It is particularly important for imbalanced classification problems. Without stratification, one subset may contain too few examples of a minority class.

The split should be randomized when observations are independent, and a random seed should be recorded for reproducibility. For time-dependent data, a chronological split is usually more appropriate than random sampling.

Cross-validation is a resampling technique used to estimate model performance and select models or hyperparameters.

In k-fold cross-validation:

1. The training data is divided into approximately equal folds.
2. The model is trained on folds and validated on the remaining fold.
3. This process is repeated times so that every fold is used once for validation.
4. The evaluation scores are averaged to obtain an overall estimate.

For example, in 5-fold cross-validation, five models are trained, each using a different fold for validation.

Advantages:

• Uses most available data for both training and validation.
• Produces a more stable estimate than a single split.
• Helps compare algorithms and tune hyperparameters.
• Reduces dependence on one particular validation sample.

Limitations:

• Requires greater computational time.
• Can be inappropriate if folds are not created according to the data structure.
• Time-series data generally requires time-aware validation.
• Preprocessing performed before splitting can cause data leakage.

For classification, stratified k-fold cross-validation is often used to preserve class proportions.

A validation set is used during model development to compare models, select features, tune hyperparameters, and decide when to stop training. A test set is held aside and used only for the final unbiased evaluation.

Differences:

• Training data is used to fit model parameters.
• Validation data is used to make development decisions.
• Test data is used to estimate final generalization performance.

If the test set is used repeatedly, developers may unconsciously adapt the model or preprocessing decisions to its observations. The test set then becomes part of the effective training process, causing an optimistic estimate of performance. This is a form of information leakage.

A proper workflow keeps the test set untouched until the pipeline and model are finalized. If data is limited, cross-validation can be used within the training data, while the test set remains reserved for the final assessment.

Data leakage occurs when information from outside the training data, especially from validation or test data, influences model training or preprocessing decisions.

Examples:

• Calculating the mean and standard deviation for scaling using the entire dataset before splitting.
• Imputing missing values using statistics calculated from both training and test records.
• Selecting features using correlations computed from the complete dataset.
• Using future information to predict a past event.
• Applying target encoding without separating the target information by fold.

Leakage produces unrealistically high validation or test performance because the model indirectly receives information about evaluation observations.

Prevention:

• Split the data before learning preprocessing parameters.
• Fit imputers, scalers, encoders, and selectors only on training data.
• Apply the fitted transformations to validation and test data without refitting.
• Use a pipeline that combines preprocessing and modeling.
• Perform all feature selection and tuning inside each cross-validation fold.

A leakage-free workflow provides a more trustworthy estimate of real-world performance.

Features can be classified according to the nature of their values.

• Numerical features: Represent measurable quantities, such as age, income, or temperature. They may be discrete or continuous. Missing values can be imputed using the mean or median, and scaling may be applied.
• Categorical nominal features: Represent categories without a meaningful order, such as color or city. One-hot encoding is commonly suitable.
• Ordinal features: Represent categories with a meaningful order, such as small, medium, and large. Ordinal encoding can preserve the order.
• Binary features: Have two possible values, such as yes/no or pass/fail. They can often be represented as 0 and 1.

Additional preprocessing may include correcting inconsistent labels, handling rare categories, removing impossible values, and addressing missing data. Choosing the correct representation is important because inappropriate numerical coding can create false relationships between categories.

Correlation and visualization can provide initial evidence about whether features may be useful for prediction.

• A correlation matrix summarizes linear relationships among numerical variables.
• A feature with very weak correlation with a numerical target may be less useful in a simple linear model, although it may still have nonlinear predictive value.
• Highly correlated input features may contain redundant information and can create multicollinearity in some models.
• Scatter plots help identify nonlinear patterns, clusters, and outliers.
• Box plots or grouped charts can show how a numerical feature differs across categories.

Precautions:

• Correlation does not prove causation.
• Pearson correlation mainly measures linear association.
• A low correlation does not imply that a feature is useless for all algorithms.
• Feature selection must not use the test set.
• Relationships should be checked with domain knowledge and suitable validation.
• Categorical variables require appropriate statistical methods or encoding.

Thus, correlation and visualization are useful screening tools, not complete substitutes for model-based evaluation.

Both normalization and standardization transform numerical features, but they use different principles.

Normalization, often referring to min-max scaling, transforms a value as:

The result commonly lies between 0 and 1. It is useful when a bounded range is required, such as in some neural network applications, or when features have known minimum and maximum values.

Standardization transforms a value as:

The transformed feature has approximately mean zero and unit standard deviation. It is commonly preferred for algorithms based on distances, gradients, or regularization when the data is not naturally bounded.

Normalization is sensitive to minimum and maximum values, especially outliers. Standardization is also affected by extreme values but is generally more flexible for unbounded data. The choice should depend on the algorithm, distribution, and domain requirements.

A target is imbalanced when one class has many more observations than another. For example, a fraud dataset may contain far more legitimate transactions than fraudulent ones.

Problems:

• A model may achieve high accuracy by predicting the majority class most of the time.
• The minority class may have poor recall and be ignored.
• A random split may place very few minority observations in one subset.

Strategies:

• Use stratified train-test splitting and stratified cross-validation.
• Evaluate with precision, recall, F1-score, balanced accuracy, ROC-AUC, or precision-recall AUC rather than accuracy alone.
• Oversample the minority class or undersample the majority class within training folds only.
• Use synthetic methods such as SMOTE when appropriate.
• Apply class weights or cost-sensitive learning.
• Adjust the decision threshold according to the application cost of errors.

The test set should retain a realistic class distribution, and resampling must not be performed before the data is split because that can cause leakage.