Unit1 - Subjective Questions

Supervised Learning:

• Definition: The model is trained on a labeled dataset, meaning each input is paired with a corresponding output label .
• Goal: To learn a mapping from inputs to outputs to predict labels for new, unseen data.
• Example: Predicting house prices based on features (Regression) or classifying emails as spam or not spam (Classification).

Unsupervised Learning:

• Definition: The model is trained on an unlabeled dataset. The system tries to learn the patterns and the structure from the data without any corresponding output variables.
• Goal: To discover underlying structures, group data, or reduce dimensionality.
• Example: Customer segmentation grouping shoppers by purchasing habits (Clustering).

Semi-Supervised Learning:

• Definition: Falls between supervised and unsupervised learning. The training dataset contains a small amount of labeled data and a large amount of unlabeled data.
• Goal: To use the unlabeled data to better capture the shape of the underlying data distribution and improve the learning accuracy of the supervised model.
• Example: Web page classification where only a few pages are manually labeled, and the algorithm uses the links and text of unlabeled pages to infer their categories.

In Unsupervised Learning, we are given a dataset consisting of observations, defined as:

where each is a -dimensional feature vector. Unlike supervised learning, there are no corresponding target labels .

The Problem Formulation:
The objective is to learn a function or a model that captures the underlying structure, distribution, or representations of the data.

Depending on the task, the formulation varies:

1. Clustering: Partition the data into distinct groups such that intra-cluster distance is minimized and inter-cluster distance is maximized.
2. Dimensionality Reduction: Find a transformation mapping to (where ) while preserving essential structural properties (e.g., variance in PCA).
3. Density Estimation: Estimate the underlying probability density function from which the data was generated.

Market Segmentation is a prime real-life use case for unsupervised learning, specifically clustering algorithms like K-Means or Hierarchical Clustering.

Process:

1. Data Collection: Businesses collect vast amounts of unlabeled data regarding customer demographics, geographic locations, purchasing history, and browsing behaviors.
2. Feature Representation: Each customer is represented as a feature vector in a multi-dimensional space.
3. Pattern Discovery: The unsupervised model groups customers into distinct clusters based on similarities in their feature vectors.
4. Actionable Insights:
   • Cluster A: High-income, frequent buyers (Target with premium products).
   • Cluster B: Price-sensitive, rare buyers (Target with discounts/coupons).

Why Unsupervised?
The company does not know the segments in advance (no labels). The algorithm organically discovers these natural groupings.

Anomaly and Fraud Detection involves identifying rare items, events, or observations that raise suspicions by differing significantly from the majority of the data.

Application of Unsupervised Learning:

1. Assumption: The majority of transactions/events are normal. Anomalies are rare and statistically different from the normal data distribution.
2. Model Training: An unsupervised algorithm (e.g., Isolation Forest, Autoencoders, or DBSCAN) is trained on historical data without explicit 'fraud' labels.
3. Detection Mechanism:
   • The model learns the standard behavior (density or clusters) of the data.
   • When a new transaction arrives, the model calculates its distance from the normal clusters or its reconstruction error.
   • If the distance or error exceeds a certain threshold , it is flagged as an anomaly.

Use Cases: Credit card fraud, network intrusion detection, and manufacturing defect detection.

Biological Data:

• Genomics: Unsupervised learning (e.g., hierarchical clustering) is used to analyze gene expression microarrays. It groups genes that have similar expression patterns across different conditions, helping identify genes that co-regulate or belong to the same biological pathway.
• Protein Sequences: Clustering algorithms help in categorizing proteins into families based on structural or sequence similarities.

Social Data:

• Social Network Analysis: Algorithms can identify 'communities' or 'cliques' within social graphs (like Twitter or Facebook) by analyzing user interactions and connections without knowing the community labels in advance.
• Trend Discovery: Topic modeling (like LDA) on social media feeds helps discover latent topics and trends being discussed in real-time without predefined categories.

Euclidean Distance is the straight-line distance between two points in Euclidean space. It is the most common metric used in clustering algorithms like K-Means.

Mathematical Formula:
For two points and in an -dimensional space, the Euclidean distance is defined as:

When to use:

• Best suited for continuous, dense data where the magnitude of the features is important.
• It works well when all dimensions are on the same scale (hence, feature scaling/standardization is crucial before using it).
• It represents the shortest physical distance, making it intuitive for spatial data.

Manhattan Distance (also known as L1 norm or City Block distance) calculates the distance between two points by summing the absolute differences of their Cartesian coordinates.

Mathematical Formula:
For two points and in an -dimensional space:

Differences from Euclidean Distance:

1. Path: Euclidean measures the shortest straight-line path, whereas Manhattan measures the path taken if one could only move along grid lines (like a taxi in Manhattan).
2. Sensitivity to Outliers: Euclidean squares the differences, making it highly sensitive to outliers. Manhattan uses absolute differences, making it more robust to outliers.
3. High Dimensionality: Manhattan distance is often preferred over Euclidean distance in high-dimensional spaces because it suffers slightly less from the curse of dimensionality.

Cosine Similarity measures the cosine of the angle between two non-zero vectors in a multi-dimensional space. It determines how similar two vectors are in terms of direction, irrespective of their magnitude.

Mathematical Formula:
For two vectors and :

Which expands to:

Significance in NLP:
In NLP, text documents are often represented as high-dimensional vectors (e.g., TF-IDF or word embeddings).

• A long document and a short document might have similar content but different vector magnitudes (frequencies of words).
• Cosine similarity evaluates the angle between these vectors, effectively ignoring the document length (magnitude). If they point in the same direction, they are considered semantically similar.

The choice of distance metric fundamentally dictates how an unsupervised algorithm perceives 'similarity' between data points, which directly impacts the resulting clusters or patterns.

Key Reasons it Matters:

1. Shape of Clusters: Euclidean distance assumes spherical clusters. If the actual data forms arbitrary shapes or grids, Manhattan or Density-based metrics might perform better.
2. Curse of Dimensionality: In very high-dimensional spaces, the Euclidean distance between the closest and farthest points tends to converge, making it useless. Metrics like Cosine or fractional distance metrics are better suited here.
3. Nature of Data:
   • Continuous/Spatial: Euclidean.
   • Text/Categorical Frequencies: Cosine Similarity.
   • Grid-like/Robustness to outliers: Manhattan.
4. Magnitude vs. Direction: If the magnitude of the vector is irrelevant (e.g., comparing user ratings where one user always rates higher than another but follows the same trend), Cosine similarity is preferred over Euclidean.

While both analyze consumer data, they focus on different aspects of the business strategy.

Market Segmentation:

• Focus: Grouping the broader market or customer base into distinct macro-segments.
• Data used: Demographics (age, location, income), broad purchasing categories.
• Output: Static or semi-static clusters (e.g., 'Millennial Tech Enthusiasts', 'Budget Shoppers').
• Goal: Broad marketing campaigns, product positioning, and brand strategy.

Customer Behavior Analysis:

• Focus: Understanding the dynamic actions, interactions, and journeys of users.
• Data used: Clickstream data, session duration, cart abandonment rates, navigation paths.
• Output: Behavioral patterns or sequences (e.g., identifying the common path users take before churning).
• Goal: UI/UX improvements, personalized real-time recommendations, and targeted interventions.

Role of Unsupervised Learning: In both cases, algorithms like K-Means or Association Rules (Apriori) find hidden structures without pre-existing labels.

Semi-Supervised Learning (SSL) leverages both labeled and unlabeled data for training—typically a small amount of labeled data and a large amount of unlabeled data.

Bridging the Gap:

• Supervised limitation: Labeling data is expensive, time-consuming, and requires domain experts.
• Unsupervised limitation: Cannot predict specific targets or categories; it only finds arbitrary structures.
• SSL Solution: SSL uses unsupervised techniques on the large unlabeled dataset to understand the fundamental data distribution and cluster boundaries. It then uses the small set of labeled data to map these clusters to specific, meaningful target variables.

Mechanisms:

• Pseudo-labeling: A model is trained on labeled data to predict labels for the unlabeled data, which are then used as 'pseudo-labels' for further training.
• Graph-based SSL: Data points are represented as nodes; edges denote similarity. Labels propagate from labeled nodes to unlabeled nodes through the edges.

Curse of Dimensionality refers to various phenomena that arise when analyzing and organizing data in high-dimensional spaces (often hundreds or thousands of dimensions).

Effect on Distance Calculations:

1. Distance Convergence: As dimensions increase, the distance between the nearest data point and the farthest data point approaches zero. Mathematically, for Euclidean distance, . This means all points appear almost equidistant from each other, making clustering meaningless.
2. Sparsity: Data points become extremely sparse in high-dimensional space. The concept of local density (used in DBSCAN) breaks down.

Mitigation:

• Use Dimensionality Reduction (PCA, t-SNE) before clustering.
• Switch to metrics less affected by volume, such as Cosine Similarity or Manhattan distance.

Impact of Feature Scaling:
Feature scaling is absolutely critical when using Euclidean distance. Since Euclidean distance calculates the squared differences between coordinates, features with larger numeric ranges will strictly dominate the distance calculation.

Justification:
Consider a dataset of humans with features: Age (range 0-100 years) and Income (range 100,000).
If we calculate the Euclidean distance without scaling:

The difference in Income (which could be in thousands) will completely overshadow the difference in Age (which is at most 100). The algorithm will essentially cluster people based only on income.

Solution: Data must be normalized (Min-Max scaling) or standardized (Z-score normalization) so all features contribute equally to the Euclidean distance.

Let and be two vectors that are -normalized, meaning their magnitudes are 1: and .

The squared Euclidean distance between and is:

Expanding this dot product:

Since the vectors are normalized, and . Substituting these in:

Since and , the Cosine Similarity is simply .
Therefore:

Conclusion: For normalized vectors, the squared Euclidean distance is inversely proportional to the Cosine similarity. Minimizing Euclidean distance is mathematically equivalent to maximizing Cosine similarity.

Distance Metric: Euclidean Distance (or Haversine formula for exact spherical earth calculations).

Why:

1. Spatial Representation: Latitude and longitude represent spatial, geographic coordinates. Euclidean distance calculates the straight-line distance between two points, which naturally aligns with physical distance in a localized 2D space.
2. Equal Weight: Both dimensions (latitude and longitude) are strictly on the same scale (degrees), meaning no single feature will artificially dominate the distance calculation without scaling.
3. Isotropic assumption: Euclidean distance treats all directions equally, meaning the distance measured North-South is equivalent to East-West, which is exactly how physical distance works.

(Note: For large, global distances, the Haversine formula is better to account for the Earth's curvature, but Euclidean is the standard generic metric for continuous spatial data).

Scenario: A routing system for delivery drones or taxicabs in an urban city built on a grid plan (like Manhattan, New York).

Justification:

• Physical Constraints: In a grid layout, vehicles cannot travel diagonally through buildings. They must move strictly along the grid lines (North/South, then East/West).
• Distance Calculation: Euclidean distance would calculate the hypotenuse (the diagonal straight line), which represents an impossible path.
• Accuracy: Manhattan distance calculates the sum of the absolute differences of their coordinates, exactly mirroring the real-world distance the vehicle must travel along the roads.

Secondary Scenario (Data Science context): When the dataset contains significant outliers. Manhattan distance (L1 norm) does not square the differences, making it much more robust against extreme outlier values compared to Euclidean distance (L2 norm).

Problem Formulation:

1. Data Representation: Create a User-Item interaction matrix where rows represent users and columns represent items (e.g., movies). The values in the matrix are the ratings given by users to items. Missing values are filled with 0 (or baseline averages).
2. Vectorization: Each user is represented as an -dimensional vector, where is the total number of items.
3. Similarity Calculation: To recommend an item to User A, the system finds other users who are similar to User A. It computes the Cosine Similarity between User A's vector and all other user vectors.
   
4. Recommendation:
   • Identify the Top-K users with the highest cosine similarity to User A.
   • Look at the items these similar users rated highly that User A has not yet seen.
   • Recommend these items to User A.

Why Cosine? It focuses on the angle (pattern of ratings). If User A rates movies (4, 4, 4) and User B rates them (2, 2, 2), they have the same taste (angle), just different rating scales (magnitudes).

Fraud detection is highly challenging when formulated as an Unsupervised Learning (anomaly detection) problem due to several factors:

1. Imbalanced Data/Extreme Rarity: Fraudulent transactions might represent less than 0.1% of total data. Unsupervised models might struggle to isolate these points without them being absorbed as 'noise' into normal clusters.
2. Dynamic Behavior (Concept Drift): Fraudsters constantly change their tactics. A pattern that is considered 'anomalous' today might become normal tomorrow, and new types of fraud won't match historical anomaly distributions.
3. Overlapping Distributions: Fraudulent activities are often specifically designed to look exactly like normal activities to avoid detection. In the feature space, the 'fraud' points heavily overlap with 'normal' points.
4. Lack of Ground Truth for Validation: Since it's unsupervised, there are no labels. It is extremely difficult to evaluate the precision and recall of the model without manual investigation of the flagged anomalies, which is time-consuming.

1. Input Data:

• Supervised: Requires labeled data. Every input vector has a corresponding ground truth label .
• Unsupervised: Requires unlabeled data. Only the input vectors are provided.

2. Expected Output:

• Supervised: A predictive model capable of mapping new inputs to continuous values (Regression) or distinct categories (Classification).
• Unsupervised: A structural model representing data groupings (Clusters), hidden patterns (Association Rules), or a compressed representation (Dimensionality Reduction).

3. Evaluation Metrics:

• Supervised: Highly objective. Uses metrics like Accuracy, Precision, Recall, F1-Score, Mean Squared Error (MSE) because the true labels are known.
• Unsupervised: Highly subjective. Uses intrinsic metrics like the Silhouette Score, Davies-Bouldin Index, or WCSS (Within-Cluster Sum of Squares) to evaluate cluster compactness and separation, as there is no absolute 'right' answer.

Overarching Goals of Unsupervised Learning:
The primary goal is to learn the underlying, hidden structure of unlabeled data. This involves discovering natural groupings, simplifying data while retaining information, and identifying rare deviations from the norm.

Mapping to Real-World Use Cases:

1. Clustering (Grouping similar data):

   • Market Segmentation: Grouping customers based on purchasing history to tailor marketing campaigns.
   • Biological Pattern Discovery: Grouping genes with similar expression levels to understand diseases.

2. Anomaly Detection (Finding outliers):

   • Fraud Detection: Identifying credit card transactions that deviate from a user's normal spending habits.
   • Predictive Maintenance: Monitoring IoT sensor data on machinery to detect unusual vibrations preceding a failure.

3. Dimensionality Reduction (Simplifying data):

   • Customer Behavior Analysis: Reducing hundreds of web interaction metrics down to 2 or 3 primary components to visualize customer journeys.
   • Data Compression: Reducing image sizes or feature counts to speed up downstream supervised learning tasks.