Unit 4: Unsupervised Learning - Subjective Questions

Unsupervised learning is a machine learning approach in which an algorithm learns patterns, structures, or relationships from data that has no predefined target labels.

Key characteristics:

• The input data is unlabeled.
• The algorithm discovers hidden patterns or groups automatically.
• It is commonly used for clustering, dimensionality reduction, and pattern discovery.

Difference from supervised learning:

• Supervised learning: Learns from labeled input-output pairs, such as predicting whether an email is spam.
• Unsupervised learning: Learns only from input data, such as grouping customers according to their purchasing behavior.

Unsupervised learning is useful when labeling data is expensive, difficult, or unavailable.

K-Means clustering partitions a dataset into clusters such that data points within the same cluster are as similar as possible.

The algorithm minimizes the within-cluster sum of squared distances:

where is cluster and is its centroid.

Major steps:

1. Select the number of clusters, .
2. Initialize centroids, usually randomly or using K-Means++.
3. Assign every data point to the nearest centroid.
4. Recalculate each centroid as the mean of the points assigned to it.
5. Repeat the assignment and update steps until the centroids stabilize or the objective function changes very little.

The final result is a set of clusters and their corresponding centroids.

For a cluster containing points , the squared-error objective with respect to centroid is:

To minimize this objective, differentiate with respect to :

Set the derivative to zero:

Therefore,

and hence,

Thus, the arithmetic mean is the point that minimizes the sum of squared distances from all points in the cluster. This result justifies the centroid update step in K-Means.

Advantages:

• Simple to understand and implement.
• Computationally efficient for large datasets.
• Works well when clusters are compact and approximately spherical.
• Produces easily interpretable cluster assignments.

Limitations:

• The number of clusters, , must be specified in advance.
• Results can depend on the initial centroid positions.
• It is sensitive to outliers because means are affected by extreme values.
• It performs poorly when clusters have irregular shapes or different densities.
• Features with larger scales can dominate the distance calculation.

Applications:

• Customer segmentation.
• Image compression and color quantization.
• Document grouping.
• Anomaly detection as a preliminary analysis.
• Grouping sensor readings according to operating conditions.

Feature scaling and suitable initialization can improve its performance.

Hierarchical clustering creates a hierarchy of nested clusters, usually represented by a dendrogram. It does not require the number of clusters to be fixed at the beginning.

Agglomerative approach:

• Starts with every data point as an individual cluster.
• Finds the two most similar clusters.
• Merges them into one cluster.
• Repeats the process until one cluster remains.

Divisive approach:

• Starts with all data points in a single cluster.
• Splits a cluster into smaller clusters.
• Continues recursively until individual points or desired groups are obtained.

Agglomerative clustering is more commonly used because it is simpler to implement. A final number of clusters can be selected by cutting the dendrogram at a suitable level.

Linkage criteria determine the distance between two clusters.

• Single linkage: Uses the minimum distance between any point in one cluster and any point in the other cluster. It can identify elongated clusters but is sensitive to chaining.
• Complete linkage: Uses the maximum pairwise distance. It tends to form compact clusters but can be affected by outliers.
• Average linkage: Uses the average distance between all pairs of points from the two clusters. It provides a compromise between single and complete linkage.
• Ward linkage: Merges the pair of clusters that causes the smallest increase in within-cluster variance. It generally produces compact and relatively balanced clusters.

The choice of linkage should depend on the expected shape of the clusters, the presence of outliers, and the distance measure used.

DBSCAN, or Density-Based Spatial Clustering of Applications with Noise, groups points according to the density of their neighborhoods.

For a point :

• Its -neighborhood contains all points within distance of .
• If this neighborhood contains at least MinPts points, is a core point.
• A point close to a core point but having fewer than MinPts neighbors is a border point.
• A point that is neither a core point nor reachable from a core point is labeled noise or an outlier.

DBSCAN expands clusters from core points using density reachability. It can discover clusters of arbitrary shape and does not require the number of clusters to be specified in advance.

K-Means is suitable for fast partitioning, hierarchical clustering is useful for understanding nested relationships, and DBSCAN is useful for noisy spatial data with irregular cluster shapes.

The number of clusters affects the quality and interpretability of K-Means results.

Elbow method:

• Run K-Means for several values of .
• Compute the within-cluster sum of squares for each value.
• Plot the error against .
• Select the point where the reduction in error begins to slow significantly. This point resembles an elbow.

Silhouette analysis:
For a point , the silhouette coefficient is:

where is the average distance from to points in its own cluster and is the smallest average distance from to another cluster.

Values close to indicate good separation, values near indicate overlapping clusters, and negative values indicate possible misclassification. The value of with a high average silhouette score is generally preferred.

Dimensionality reduction is the process of representing data with fewer variables while preserving the most important information.

Importance:

• Reduces computational cost.
• Decreases storage requirements.
• Removes redundant or noisy features.
• Helps prevent the curse of dimensionality.
• Enables visualization of high-dimensional data in two or three dimensions.
• Can improve the performance of clustering algorithms.

Feature selection:

• Selects a subset of the original features.
• The selected features retain their original meaning.
• Examples include filter, wrapper, and embedded methods.

Feature extraction:

• Creates new features by transforming or combining the original features.
• The new features may be less directly interpretable.
• Principal Component Analysis is a common feature extraction technique.

Principal Component Analysis, or PCA, transforms correlated variables into a smaller set of uncorrelated variables called principal components.

Steps:

1. Standardize the features when they have different units or scales.
2. Compute the covariance or correlation matrix.
3. Calculate the eigenvalues and eigenvectors of the matrix.
4. Sort the eigenvectors in descending order of their eigenvalues.
5. Select the first few eigenvectors as principal components.
6. Project the original data onto the selected components.

The first principal component captures the maximum possible variance. Each subsequent component captures the maximum remaining variance while being orthogonal to the previous components. PCA therefore provides a lower-dimensional representation with minimal information loss based on variance.

Assume the centered data matrix is and is a unit vector representing a possible projection direction. The projected data is:

The variance of the projected data is proportional to:

where is the covariance matrix of the data. We want to maximize this variance subject to :

Using a Lagrange multiplier :

Differentiating and setting the result to zero gives:

Thus, must be an eigenvector of the covariance matrix. The direction with the largest eigenvalue gives the first principal component because it captures the greatest variance.

Advantages:

• Reduces the number of features while retaining important variance.
• Removes linear correlations among transformed features.
• Improves visualization of high-dimensional data.
• Can reduce noise and computational cost.
• Often improves the efficiency of clustering and classification pipelines.

Limitations:

• Principal components are linear combinations and may be difficult to interpret.
• PCA is sensitive to feature scaling.
• It captures variance, not necessarily the information most useful for a particular task.
• It may perform poorly when relationships are nonlinear.
• It can be sensitive to outliers.
• Choosing the number of components involves a trade-off between compression and information retention.

PCA should be applied carefully, especially when interpretability or nonlinear structure is important.

Association rule mining discovers relationships among items or events in a dataset. A rule is commonly written as , where is the antecedent and is the consequent.

Support: Measures how frequently and occur together.

Confidence: Measures how often occurs when occurs.

Lift: Compares the confidence of the rule with the independent occurrence of .

A lift greater than suggests a positive association, a lift equal to suggests independence, and a lift less than suggests a negative association.

The Apriori algorithm identifies frequent itemsets and generates association rules from transaction data.

Main steps:

1. Find frequent one-itemsets using a minimum support threshold.
2. Generate candidate itemsets of larger size from previously identified frequent itemsets.
3. Prune candidates whose subsets are not frequent.
4. Scan the database to calculate candidate support.
5. Keep candidates meeting the minimum support requirement.
6. Repeat until no further frequent itemsets can be generated.
7. Generate rules from frequent itemsets and retain rules satisfying minimum confidence.

The downward closure principle states that every subset of a frequent itemset must also be frequent. Equivalently, if an itemset is infrequent, all of its supersets must be infrequent. This property substantially reduces the search space.

There are four transactions in total.

The itemset occurs in two transactions: and .

The item occurs in three transactions, and item occurs in three transactions.

Support:

Confidence:

Lift:

The confidence is approximately . Since the lift is less than , the occurrence of does not indicate a positive association with in this dataset.

Frequent itemset generation:

• Finds groups of items that occur together frequently.
• Uses the minimum support threshold.
• Produces itemsets such as or .
• Uses the downward closure property to prune candidates.

Association rule generation:

• Converts frequent itemsets into directional rules.
• Splits an itemset into antecedent and consequent parts.
• Uses confidence and other measures to evaluate the rules.
• Produces rules such as .

For example, if is frequent, possible rules include , , and . Only rules meeting the minimum confidence requirement are retained.

Unsupervised learning helps robots learn useful structure from sensor and operational data without requiring manual labels.

Perception:

• Cluster visual features or point-cloud observations to identify similar objects or regions.
• Use dimensionality reduction to represent images or sensor measurements compactly.

Navigation:

• Group spatial observations into regions with similar properties.
• Detect changes in environmental patterns and identify unexplored areas.
• Discover recurring movement or obstacle configurations.

Fault detection:

• Learn clusters representing normal operating states.
• Identify sensor readings that do not belong to any normal cluster.
• Use association rules to discover relationships among motor temperature, vibration, current, and speed.

For example, DBSCAN can label unusual lidar points as noise, while PCA can reduce the dimensionality of a robot's multi-sensor state vector before clustering.

Robots generate high-dimensional sensor data from cameras, lidar, accelerometers, gyroscopes, motors, and environmental sensors.

Role of dimensionality reduction:

• Compresses correlated sensor measurements into a smaller number of components.
• Removes redundancy and some noise.
• Makes visualization and real-time processing easier.
• Reduces the memory and computational requirements of later algorithms.

Role of clustering:

• Groups sensor readings representing similar operating states.
• Identifies behaviors such as idle, moving, turning, climbing, or carrying a load.
• Detects abnormal states when new readings do not match known clusters.

A common pipeline is to clean and scale the sensor data, apply PCA, and then use K-Means or DBSCAN. The resulting clusters can support behavior analysis, predictive maintenance, and autonomous decision-making.

Challenges and solutions:

• Noise and outliers: Use filtering, robust scaling, DBSCAN, or explicit outlier detection.
• Different measurement scales: Standardize or normalize features before distance-based clustering.
• High dimensionality: Apply PCA or another dimensionality reduction method.
• Changing environments: Use incremental learning or periodically update the model.
• Unequal cluster densities: Tune DBSCAN carefully or use density methods that support varying densities.
• Unknown number of behaviors: Compare multiple values of using validation measures such as silhouette score.
• Limited interpretability: Examine representative samples and relate clusters to physical robot states.
• Real-time constraints: Use efficient algorithms, reduced feature sets, and incremental processing.

Reliable deployment also requires sensor synchronization, missing-value handling, and evaluation by domain experts.