Unit 4: Machine learning-1 - Subjective Questions

Machine Learning (ML) is a subfield of Artificial Intelligence that enables computer systems to automatically learn patterns from data and improve their performance on a task without being explicitly programmed for every scenario.

The three broad categories are:

• Supervised Learning: The model is trained on a labeled dataset where both input features and the correct output are provided. The goal is to learn a mapping function .

  • Examples: Spam detection (classification), house price prediction (regression).

• Unsupervised Learning: The model works with unlabeled data and tries to discover hidden structures or patterns.

  • Examples: Customer segmentation (clustering), dimensionality reduction (PCA).

• Reinforcement Learning: An agent learns by interacting with an environment and receiving rewards or penalties, aiming to maximize cumulative reward.

  • Examples: Game playing (AlphaGo), robotics navigation.

The key differences are:

Summary:

• Supervised learning learns from examples with answers.
• Unsupervised learning learns from data without answers, grouping or reducing based on similarity.

Both are supervised learning tasks that differ by the nature of the output variable.

Classification:

• The output variable is categorical/discrete (belongs to a class or category).
• Goal: Assign inputs to predefined classes.
• Examples:
  • Email as spam or not spam
  • Tumor as benign or malignant
• Algorithms: Logistic Regression, Decision Trees, SVM, KNN.

Regression:

• The output variable is continuous/numeric.
• Goal: Predict a real-valued quantity.
• Examples:
  • Predicting house prices
  • Forecasting temperature
• Algorithms: Linear Regression, Polynomial Regression, Random Forest Regressor.

Key Difference: Classification predicts a label, whereas regression predicts a quantity.

A typical supervised ML workflow consists of the following stages:

1. Problem Definition: Clearly identify the task (classification/regression) and objective.

2. Data Collection: Gather labeled data relevant to the problem.

3. Data Preprocessing:

   • Handle missing values
   • Encode categorical variables
   • Normalize/standardize features
   • Remove outliers

4. Data Splitting: Divide into training, validation, and test sets (e.g., 70/15/15).

5. Model Selection: Choose an appropriate algorithm (e.g., Logistic Regression).

6. Training: Fit the model on the training data by optimizing parameters.

7. Evaluation: Assess performance using metrics like accuracy, precision, recall, or RMSE on the test set.

8. Hyperparameter Tuning: Optimize model parameters using validation set.

9. Deployment: Integrate the model into a production environment.

10. Monitoring: Continuously track performance and retrain as needed.

Logistic Regression is a supervised learning algorithm used for binary classification problems. It models the probability that a given input belongs to a particular class.

Instead of fitting a straight line like linear regression, it applies the sigmoid (logistic) function to map any real-valued input into the range :

where .

Why it is called regression:

• It uses a regression equation (linear combination of inputs) internally to compute .
• However, the output is passed through the sigmoid function to produce a probability, which is then thresholded (e.g., at ) to assign a class.

Why used for classification:

• The output represents , a probability.
• A decision boundary is created to separate classes.
• Thus, although mathematically a regression model, its final purpose is classification.

The Sigmoid (logistic) function is defined as:

where is the linear combination of features.

Derivation of its odds interpretation:

Starting from the probability :

Rearranging:

Taking natural log gives the log-odds (logit):

Important Properties:

• Range: Output always lies in , suitable for probabilities.
• S-shaped curve: Smooth and monotonically increasing.
• Symmetry: .
• Derivative: , which simplifies gradient computation.
• At , — the natural decision threshold.

Logistic regression uses the Log Loss (Binary Cross-Entropy) cost function:

where:

• is the predicted probability.
• is the actual label (0 or 1).
• is the number of training examples.

Interpretation:

• If , cost — penalizes low predicted probabilities.
• If , cost — penalizes high predicted probabilities.

Why MSE is NOT preferred:

• When the sigmoid function is combined with Mean Squared Error, the resulting cost function becomes non-convex (many local minima).
• This makes gradient descent likely to get stuck in local minima.
• Log Loss, on the other hand, produces a convex cost surface, guaranteeing convergence to the global minimum.

Gradient Descent is an iterative optimization algorithm used to minimize a cost function by updating parameters in the direction of the steepest descent (negative gradient).

Update Rule:

where is the learning rate.

For Logistic Regression, the gradient of the log-loss cost function is:

So the update becomes:

Steps:

1. Initialize parameters (often zeros).
2. Compute predictions using the sigmoid function.
3. Calculate the cost and gradient.
4. Update parameters using the update rule.
5. Repeat until convergence.

Role of Learning Rate ():

• Too small → slow convergence.
• Too large → may overshoot or diverge.

A Decision Boundary is the surface that separates the feature space into regions corresponding to different predicted classes.

In logistic regression, the model predicts class when:

Since when , the decision boundary is given by:

Linear Decision Boundary:

• When features enter linearly, e.g., .
• Produces a straight line (or hyperplane) separating classes.

Non-linear Decision Boundary:

• By adding polynomial features, e.g., .
• Produces curved boundaries (circles, ellipses) enabling separation of complex data.

Key Point: The decision boundary is a property of the model parameters, not the training data itself.

Clustering is an unsupervised learning technique that groups similar data points together based on feature similarity, without using labeled outputs.

K-Means Clustering partitions data into clusters, each represented by a centroid.

Algorithm Steps:

1. Choose K: Decide the number of clusters .

2. Initialize Centroids: Randomly select points as initial cluster centers.

3. Assignment Step: Assign each data point to the nearest centroid using a distance metric (usually Euclidean):

4. Update Step: Recompute each centroid as the mean of all points assigned to it:

5. Repeat: Iterate the assignment and update steps until centroids no longer change significantly (convergence).

Objective: Minimize the within-cluster sum of squares (WCSS):

Summary: Linear regression estimates a quantity, while logistic regression estimates the probability of a category.

Logistic regression can be extended based on the number and nature of output categories:

1. Binary Logistic Regression:

• The target variable has exactly two classes (0 or 1).
• Example: Email → spam / not spam.
• Uses the standard sigmoid function.

2. Multinomial Logistic Regression:

• The target has more than two unordered categories.
• Example: Classifying fruit as apple, banana, or orange.
• Uses the Softmax function:

3. Ordinal Logistic Regression:

• The target has more than two ordered categories.
• Example: Customer rating → poor < average < good < excellent.
• Considers the order/ranking among categories.

Key Distinction: Binary handles 2 classes; multinomial handles multiple unordered classes; ordinal handles multiple ordered classes.

A Confusion Matrix is a table used to evaluate the performance of a classification model by comparing predicted vs actual labels.

Key Metrics:

• Accuracy: Fraction of correct predictions.
  

• Precision: Of predicted positives, how many are correct.
  

• Recall (Sensitivity): Of actual positives, how many detected.
  

• F1-Score: Harmonic mean of precision and recall.
  

Note: Accuracy alone is misleading for imbalanced datasets; precision, recall, and F1-score give better insight.

Overfitting:

• The model learns the training data too well, including noise and outliers.
• Results in high accuracy on training data but poor performance on new/test data.
• Indicates high variance.

Underfitting:

• The model is too simple to capture the underlying patterns.
• Results in poor performance on both training and test data.
• Indicates high bias.

Remedies for Overfitting:

• Use regularization (L1/L2).
• Collect more training data.
• Reduce model complexity / feature count.
• Apply cross-validation.
• Use dropout (in neural networks).

Remedies for Underfitting:

• Increase model complexity.
• Add more relevant features.
• Reduce regularization strength.
• Train longer.

Goal: Achieve a good bias-variance tradeoff for optimal generalization.

Regularization is a technique used to prevent overfitting by adding a penalty term to the cost function that discourages large parameter values.

Regularized Cost Function (Logistic Regression):

where is the regularization parameter.

L1 Regularization (Lasso):

• Adds penalty proportional to absolute values of coefficients: .
• Can shrink some coefficients to exactly zero → performs feature selection.
• Produces sparse models.

L2 Regularization (Ridge):

• Adds penalty proportional to squared values: .
• Shrinks coefficients toward zero but not exactly zero.
• Handles multicollinearity well.

Effect of :

• Large → more regularization → risk of underfitting.
• Small → less regularization → risk of overfitting.

Applications of Supervised Learning:

• Spam Detection: Classifying emails as spam or not.
• Medical Diagnosis: Predicting disease presence from patient data.
• Credit Scoring: Assessing loan default risk.
• Image Recognition: Identifying objects/faces in images.
• Sales Forecasting: Predicting future sales (regression).
• Sentiment Analysis: Classifying text as positive/negative.

Applications of Unsupervised Learning:

• Customer Segmentation: Grouping customers by buying behavior.
• Anomaly Detection: Detecting fraud or network intrusions.
• Recommendation Systems: Grouping similar products/users.
• Market Basket Analysis: Finding item associations.
• Dimensionality Reduction: Compressing features via PCA.
• Document Clustering: Organizing large text collections.

Summary: Supervised learning excels where labeled data and prediction are needed; unsupervised learning excels in discovering hidden structure in unlabeled data.

Maximum Likelihood Estimation (MLE) is the method used to estimate the parameters of a logistic regression model by finding values that maximize the likelihood of observing the given training data.

Likelihood Function:

For independent observations with :

Log-Likelihood (easier to optimize):

Key Points:

• Maximizing the log-likelihood is equivalent to minimizing the negative log-likelihood (log-loss cost function).
• There is no closed-form solution, so iterative methods like Gradient Ascent/Descent or Newton's method are used.
• MLE gives the best-fitting probability model for the observed data.

Key Insight:

• In classification, the categories are known in advance.
• In clustering, the groups are discovered from the data based on similarity, and their meaning is interpreted afterward.

Dimensionality Reduction is an unsupervised learning technique that reduces the number of input features (dimensions) while preserving as much important information as possible. It helps in:

• Reducing computational cost
• Removing noise and redundancy
• Avoiding the curse of dimensionality
• Enabling visualization of high-dimensional data

Principal Component Analysis (PCA):

PCA transforms correlated features into a smaller set of uncorrelated variables called principal components, ordered by the amount of variance they capture.

Steps:

1. Standardize the data.
2. Compute the covariance matrix of features.
3. Calculate eigenvalues and eigenvectors of the covariance matrix.
4. Sort eigenvectors by decreasing eigenvalues.
5. Select top eigenvectors to form the projection matrix.
6. Transform data into the new reduced-dimensional space.

Key Idea: The first principal component captures the maximum variance, the second captures the next maximum (orthogonal to the first), and so on.

Given:

Step 1 — Apply the Sigmoid Function:

Since :

Step 2 — Apply the Threshold ():

Since predicted probability , the model predicts Class 1 (Positive).

Step 3 — Interpretation:

• The model estimates about 88% probability that the input belongs to the positive class.
• Because this exceeds the decision threshold of , the final classification is positive.
• A positive value always yields probability , while a negative yields probability .