Unit 6: Unsupervised Learning - Practice Quiz

Unsupervised learning works with data that does not have predefined labels or target outcomes, aiming to discover underlying patterns, groupings, or structures on its own.

Supervised learning requires a dataset with labeled outputs (or target variables) for training, whereas unsupervised learning works with unlabeled data.

Euclidean distance is the length of the straight line connecting two points. For two points and , it is calculated as .

Manhattan distance, also known as L1 norm or taxicab distance, measures distance by summing the absolute differences between the coordinates of two points, as if navigating a city grid.

Cosine similarity measures the cosine of the angle between two vectors. A value of 1 means the angle is 0 degrees, so the vectors point in the exact same direction, regardless of their magnitude.

Clustering is the task of partitioning a dataset into groups (clusters) such that points within a single cluster are highly similar, and points in different clusters are dissimilar.

The 'K' in K-Means is a hyperparameter that specifies the desired number of clusters the algorithm should partition the data into.

The 'Means' in K-Means refers to this step: after assigning all points to the nearest centroid, each centroid is moved to the mean (average) position of all the points in its cluster.

The Elbow method involves running K-Means for a range of K values and plotting the inertia (sum of squared distances). The 'elbow' point on the plot, where the rate of decrease sharply changes, suggests a good value for K.

K-Medoids is a variation of K-Means where the center of each cluster (the medoid) is restricted to be one of the actual data points in the dataset. This makes it more robust to outliers than K-Means.

A dendrogram is a tree-like diagram that records the sequence of merges or splits, showing the hierarchical relationship between clusters.

Agglomerative clustering begins with each data point as its own cluster and iteratively merges the closest pair of clusters until only one cluster (containing all points) remains.

Complete linkage, also known as the farthest-neighbor method, defines the distance between two clusters as the longest possible distance between any two points belonging to those two clusters.

Unlike K-Means, which assumes clusters are spherical, density-based algorithms like DBSCAN can identify clusters of non-linear and arbitrary shapes by connecting points in high-density regions.

Anomaly detection (or outlier detection) is the process of finding rare items, events, or observations which raise suspicions by differing significantly from the majority of the data.

Inertia, also known as within-cluster sum-of-squares (WCSS), measures how internally coherent the clusters are. A lower inertia value indicates that the clusters are more dense and compact.

A high Silhouette Score (near +1) signifies that the point is far from neighboring clusters and close to other points in its own cluster, indicating a good clustering assignment for that point.

A Silhouette Score around 0 indicates that the point is on or very close to the decision boundary, meaning its distance to its own cluster and the neighboring cluster is very similar.

The Davies–Bouldin Index measures the average similarity ratio of each cluster with its most similar cluster. Lower values indicate better clustering, where clusters are compact (low intra-cluster distance) and well-separated (high inter-cluster distance).

In high-dimensional spaces like text analysis (where each word is a dimension), the magnitude of vectors (e.g., document length) can be misleading. Cosine similarity focuses on the orientation (i.e., the relative frequency of words), which is often more relevant for determining topic similarity.

Unsupervised learning is used when there are no explicit labels in the data. The objective is to infer the natural structure present within a dataset. Customer segmentation is a classic example of clustering, a type of unsupervised learning, where the algorithm groups similar customers together without any prior knowledge of the group definitions.

The Euclidean distance is calculated as . The Manhattan distance is calculated as .

The K-Means algorithm is an iterative two-step process. The first step is the 'assignment step,' where each data point is assigned to its nearest centroid. The second step is the 'update step,' where the centroid of each cluster is moved to the mean (center) of all the data points that have been assigned to it. This cycle repeats until convergence.

The Elbow method aims to find the point where the rate of decrease in inertia slows down significantly, forming an 'elbow' shape in the plot. This point represents a trade-off where increasing the number of clusters (K) beyond this point does not significantly explain more of the variance in the data. Choosing K=10 would lead to overfitting, as inertia will always decrease as K increases.

In K-Means, the centroid is the mean of the points in a cluster, which can be heavily skewed by outliers. In K-Medoids, the representative point (medoid) is restricted to be one of the actual data points in the cluster. This constraint makes the medoid less sensitive to the influence of extreme outlier values, resulting in more robust clusters.

A dendrogram visually represents the process of hierarchical clustering. The vertical lines represent data points or clusters. By cutting the dendrogram with a horizontal line, you are setting a dissimilarity threshold. The number of vertical lines the horizontal line crosses determines the number of clusters formed at that threshold. It provides a way to partition the data into a specific number of clusters, but doesn't necessarily mean that number is optimal.

Single linkage measures the distance between the two closest points of two clusters. This can lead to a 'chaining effect' where clusters are merged if even a single pair of points is close, regardless of the overall cluster shapes. This makes it sensitive to noise and often results in elongated, non-compact clusters.

In DBSCAN, a point is a 'core point' if it has at least MinPts neighbors (including itself) within the eps-radius. Point A has 3 neighbors, so it only has 4 points in its neighborhood, which is less than MinPts=5. Therefore, it is not a core point. A 'border point' is not a core point but lies within the eps-radius of a core point. Since the question states it's not in the radius of any core point, it cannot be a border point. Any point that is neither a core nor a border point is classified as a 'noise point'.

The Silhouette Score ranges from -1 to 1. A score close to +1 indicates that the object is well matched to its own cluster and poorly matched to neighboring clusters. A score close to 0 indicates that the object is on or very close to the decision boundary between two neighboring clusters. A negative score (close to -1) indicates that the object has likely been assigned to the wrong cluster, as it is closer to the points in a neighboring cluster than to the points in its own cluster. An average score of -0.2 suggests a poor clustering configuration.

The Davies-Bouldin Index is an evaluation metric where a lower value signifies a better clustering. It is calculated as the average similarity measure of each cluster with its most similar cluster. 'Similarity' is the ratio of within-cluster distances to between-cluster distances. Therefore, a lower DBI score indicates that clusters are more compact (low within-cluster distance) and well-separated (high between-cluster distance). In this case, 0.75 is lower than 1.2, so K=3 provides a better clustering result according to DBI.

Standard distance metrics like Euclidean or Manhattan are designed for numerical data and perform poorly with mixed data types. Cosine distance is typically for high-dimensional text data. Gower's Distance is specifically designed for mixed data types. It computes a weighted sum of distances for each variable type: it uses range-normalized Manhattan distance for numerical features and Dice's coefficient or Jaccard index for categorical features, making it ideal for this kind of heterogeneous dataset.

Unsupervised anomaly detection is about finding data points that deviate significantly from the majority of the data, assuming no prior labels for what constitutes an 'anomaly'. Credit card fraud detection fits this perfectly, as the goal is to flag rare, suspicious activities that don't conform to normal spending patterns. Spam classification is supervised learning, stock price prediction is a regression problem (supervised), and grouping articles is a clustering task.

Inertia measures how internally coherent clusters are (i.e., how close points are to their centroid). However, it is a decreasing function of the number of clusters, K. In the extreme case where K equals the number of data points, inertia will be 0. This makes it impossible to use inertia alone to choose K, as it will always favor the highest value of K. Metrics like the Silhouette Score or Davies-Bouldin Index are needed because they balance intra-cluster cohesion with inter-cluster separation.

Vector V1 is exactly 5 times vector V2 (5 * [2, 4, 1] = [10, 20, 5]). Cosine similarity measures the cosine of the angle between two vectors. Since V1 and V2 are scalar multiples, they are collinear and point in the same direction, so the angle between them is 0 degrees. The cosine of 0 is 1. Cosine Distance is calculated as 1 - Cosine Similarity. Therefore, the distance is 1 - 1 = 0. This demonstrates that cosine distance is insensitive to vector magnitude and only considers orientation, which is useful in applications like text analysis where document length can vary.

Principal Component Analysis (PCA) is an unsupervised learning technique. It transforms the data into a new coordinate system of principal components without using any labels. Its primary use is for dimensionality reduction, where it identifies the directions (components) that capture the most variance in the data, allowing the original feature space to be represented by a smaller number of components. This is a common preprocessing step before applying supervised algorithms.

K-Means partitions data by minimizing the sum of squared distances to a cluster's centroid. This objective function inherently assumes that clusters are spherical (isotropic) and convex. It struggles with elongated, non-convex, or irregularly shaped clusters. DBSCAN defines clusters as dense regions of points, allowing it to identify clusters of any shape as long as the density criterion is met. While DBSCAN also doesn't require K to be specified, its main advantage in finding arbitrary shapes comes from dropping the convexity/sphericity assumption.

The silhouette plot visualizes the silhouette score for each sample, grouped by cluster. A low average score for a cluster, especially with many negative values, indicates that the cluster is not well-separated. Negative scores for individual points mean they are, on average, closer to the points in a different cluster than to the points in their own cluster. This suggests that the cluster may not be a natural grouping or that the overall K value may be incorrect.

K-Means is a hill-climbing algorithm that is sensitive to the initial placement of centroids. If the initial centroids are not well-placed, the algorithm can converge to a local minimum of the inertia function, which is not the globally optimal solution. Placing two centroids within one natural cluster will likely cause the algorithm to partition that cluster into two smaller, artificial clusters, as each centroid carves out its own Voronoi cell from the surrounding points.

For datasets with well-separated, spherical (globular) clusters, both Complete linkage and Ward's linkage tend to perform well. Complete linkage is less susceptible to noise than single linkage and produces more compact clusters. Ward's linkage specifically aims to minimize the total within-cluster variance, which naturally favors finding spherical clusters. Single linkage would be a poor choice as it can create long chains. In this ideal scenario, the results from Complete and Ward's linkage would likely be very similar and effective.

High-dimensional and sparse data, such as text or purchase history, often suffers from the 'curse of dimensionality,' where Euclidean and Manhattan distances become less meaningful. Cosine distance is ideal for this type of data because it measures the orientation (angle) of the vectors rather than their magnitude. It effectively captures the similarity of purchase patterns regardless of the total number of items purchased. K-Means (or its variant Spherical K-Means) is frequently used with cosine distance for such problems.

The geometric mean minimizes the sum of squared Euclidean distances, which is the standard K-Means objective. The geometric median, however, is the point that minimizes the sum of Euclidean distances to all points in a set. This new objective function is less sensitive to the large errors caused by outliers (since the distances are not squared), making the resulting algorithm (known as K-medians clustering, not to be confused with K-Medoids) more robust. Option A is incorrect because this describes using Manhattan distance and component-wise medians. Option C is incorrect; while it's no longer minimizing WCSS, the algorithm would still converge to a local minimum of the new objective function. Option D is incorrect as the geometric median is not necessarily one of the data points, which is the defining characteristic of a medoid.

A naive implementation of agglomerative clustering requires computing and storing a proximity matrix of size , which for would require an unfeasibly large amount of memory ( space). All standard linkage methods (single, complete, average, Ward's) face this bottleneck. The most effective and common strategy for large datasets is a hybrid approach, like the one described. First, partition the data into a larger number of small clusters (micro-clusters) using a scalable algorithm like K-Means or BIRCH. Then, perform the more computationally expensive hierarchical clustering on the centroids of these micro-clusters. This drastically reduces for the hierarchical step, making the problem tractable.

DBSCAN uses a single, global set of parameters (eps and min_samples) to define density. To identify the very dense Cluster A, you would need a small eps and/or a high min_samples. When these parameters are applied to the much sparser Cluster B, it's highly probable that none of its points will qualify as core points (i.e., they won't have min_samples neighbors within the small eps radius). As a result, the points in Cluster B, which do not meet the high-density criteria set for Cluster A, will be classified as noise. This highlights a key limitation of DBSCAN, which is addressed by algorithms like OPTICS.

The Silhouette Score for a single point, , ranges from -1 to 1. A score near 0 indicates overlapping clusters. A negative score means that , where is the mean intra-cluster distance and is the mean nearest-cluster distance. A negative score for a point implies it is, on average, closer to the points in a neighboring cluster than to the points in its own cluster. An average score of -0.1 means that this is true for a significant portion of the dataset, indicating a very poor clustering where many points are misassigned, likely due to heavy overlap or an incorrect cluster structure being imposed on the data.

This is a classic problem in collaborative filtering. In a user-movie matrix, a missing rating is usually represented as a 0. For two users who have rated completely different sets of movies, their vectors will be orthogonal but far apart in Euclidean space. For example, User A = [5, 5, 0, 0] and User B = [0, 0, 5, 5]. Euclidean distance between them will be large, suggesting they are dissimilar. However, they haven't expressed disagreement; they've just rated different things. Cosine similarity measures the angle between vectors, ignoring magnitude. It correctly identifies that we don't have enough information to judge their similarity or dissimilarity based on these ratings. Crucially, Euclidean distance treats the shared zeros (unrated movies) as a point of agreement on a low rating, which is semantically incorrect and biases the similarity measure.

The Davies-Bouldin Index for a cluster is calculated by finding , where ( = intra-cluster scatter, = inter-cluster distance). The final DBI is the average of these maximums over all clusters. If you have a large, sparse cluster (high ) and a small, dense cluster (low ), the term can be large. Even if the distance between their centroids () is also large, the ratio can be high, leading to a poor DBI score for the larger cluster. This sensitivity to variations in cluster density and size can make the DBI misleading in such scenarios, even when the clusters are visually distinct.

The Elbow method relies on the principle that adding another cluster will significantly reduce WCSS if it splits a natural cluster, but the reduction will be marginal if it splits a homogeneous group. If the data has no real underlying cluster structure and is just a single blob (like a Gaussian distribution), then each additional centroid added by increasing will simply find a new pocket of variance to explain. This results in a steady, gradual decrease in WCSS without any sharp drop or 'elbow' point. Uniform data might also produce a smooth curve. In contrast, well-separated clusters would create a very sharp elbow, and non-spherical clusters would still likely show some (less clear) elbow, but the underlying WCSS metric itself would be inappropriate.

In K-Means, updating the centroid is a simple operation (calculating the mean of the points in the cluster). In the swap step of the PAM algorithm, for each cluster, it iteratively considers swapping the current medoid with every other non-medoid point within that cluster to see if the total cost decreases. This makes the update step for a single iteration have a complexity of approximately , which is much higher than K-Means' . This computational burden is the primary trade-off for the increased robustness and flexibility of K-Medoids.

Inertia (or WCSS) is the specific objective function that K-Means aims to minimize. A lower inertia for the same dataset and same means the algorithm found a solution that better minimizes this specific mathematical objective. Since K-Means with random initializations is prone to converging to local minima, the most direct and certain conclusion is that Run A found a better solution in terms of the WCSS objective than Run B. We cannot conclude that Run A is 'qualitatively better' (Option A) because minimizing WCSS is not always aligned with the desired real-world outcome, especially for non-spherical clusters. We also cannot conclude anything about the shape or size of the clusters (Option C), only the sum of squared distances. Option D is contradicted by the problem statement that data was standardized for both.

DBSCAN identifies anomalies as points that are not part of any dense region (noise points). Its fundamental assumption for anomaly detection is that anomalies are isolated points. If a set of anomalous events are similar to each other, they might form their own small, dense cluster. If this new cluster has enough points (min_samples) within its eps radius, DBSCAN will identify it as a valid, normal cluster rather than flagging its members as anomalies. This would result in false negatives. This scenario is known as 'micro-cluster' anomaly, and it's a known weakness of using standard density-based clustering for anomaly detection.

This question highlights how different distance metrics perceive space.

1. Euclidean Distance (shortest direct path):
   
   
   Under Euclidean distance, A is closer to P.
2. Manhattan Distance (city block distance):
   
   
   Under Manhattan distance, B is closer to P.
   This demonstrates that the notion of 'closeness' is dependent on the metric used, which is critical when choosing algorithms for grid-like or feature-independent spaces (favoring Manhattan) vs. continuous, feature-dependent spaces (favoring Euclidean).

Single linkage defines the distance between two clusters as the minimum distance between any two points in the respective clusters. This property makes it highly susceptible to outliers. A single outlier located between two otherwise separate clusters can create a 'bridge,' causing the algorithm to merge them prematurely. This same property leads to the 'chaining' phenomenon, where the algorithm keeps adding points to the end of an existing cluster, creating long, non-compact shapes. Complete linkage is the opposite; it's sensitive to outliers but tends to produce compact, spherical clusters because it's based on the maximum distance.

This question touches the boundary between unsupervised and supervised learning. The core issue is the mismatch of objectives. K-Means finds groups based on feature similarity (low variance). Fraudulent transactions are not necessarily 'similar' to each other in a way that forms a neat, low-variance cluster. Conversely, there might be many types of legitimate but rare transaction patterns (e.g., buying a rare collectible, a one-time international purchase) that would form small, isolated clusters. The K-Means algorithm, in its quest to minimize variance, might flag these legitimate rare events as anomalies and fail to group different types of fraud together. The fundamental assumption that 'fraud forms its own clusters' may not hold, and the algorithm's objective is not trained to find the specific patterns that define fraud.

While using clustering for feature engineering can be powerful, this specific implementation is flawed. The K-Means algorithm assigns cluster labels (0, 1, 2,...) arbitrarily. There is no inherent meaning to the ordering; cluster 5 is not 'greater than' or 'more of something' than cluster 4. By encoding this as a single numerical feature, most supervised models (like linear regression, SVMs, or gradient boosting trees) will interpret this as an ordinal variable, assuming a monotonic relationship that does not exist. This misleads the model and damages its performance. The correct way to use these cluster assignments as features would be to one-hot encode them, creating 20 new binary features, which treats them as distinct categories without an artificial order.

This scenario is a classic use-case for density-based clustering. K-Means and Ward's linkage both have strong assumptions that clusters are convex or spherical and will fail catastrophically on interlocking crescent shapes, likely splitting each crescent and grouping parts from both. GMMs also assume clusters are Gaussian (ellipsoidal), which is a poor fit for crescents. DBSCAN, however, defines clusters as areas of high density. It can trace the arbitrary shape of the crescents as long as their points are density-connected. By carefully tuning eps to the 'thickness' of the crescents and min_samples to be higher than the density of the surrounding noise, it can successfully identify the two clusters and explicitly classify the noisy points as a separate 'noise' category.

Inertia will almost always decrease as increases, because adding more centroids allows the algorithm to reduce the total sum of squared distances. It is a measure of cluster compactness only. The Silhouette Score, however, measures both compactness () and separation (). By setting , the algorithm was forced to split one of the natural clusters from the solution. While this split reduced the overall Inertia (as points are now closer to their new centroids), it created two new, less-separated clusters. Points along the split boundary now have a high intra-cluster distance () and a low nearest-cluster distance (), leading to a poor Silhouette Score for those points and pulling the overall average down. This divergence is a classic sign of over-clustering.

The key innovation of OPTICS (Ordering Points To Identify the Clustering Structure) is that it doesn't assign cluster membership directly for a single eps value. Instead, it produces an ordering of the data points and their 'reachability distance'. This ordering places points that are close in density together. The resulting reachability plot can be visualized as a terrain map with valleys representing dense clusters. By setting a threshold on this plot, one can extract clusters equivalent to what DBSCAN would find for a specific eps. More importantly, one can use techniques to find the valleys automatically, allowing it to identify clusters of different densities from a single run of the algorithm, which is something DBSCAN cannot do.

Inertia is the sum of squared Euclidean distances. Euclidean distance is sensitive to the scale of the features. A feature like income, with a range in the tens of thousands, will contribute much more to the squared distance calculation than a feature like age, with a range of less than 100. Consequently, the K-Means algorithm, in its effort to minimize this distance-based objective function, will prioritize minimizing the variance of the high-scale feature. The resulting centroids and cluster assignments will be almost entirely determined by the 'income' feature, and the clusters will appear as vertical or horizontal bands when plotted, effectively ignoring the information in the lower-scale features.

Isolation Forest works by building random trees. The core idea is that an anomalous point is easier to 'isolate' from the rest of the data than a normal point. In a tree structure, isolating a point means creating a path from the root to a leaf that contains only that point. Because anomalies are 'different', a random split is more likely to separate an anomaly from the rest of the data early on, resulting in a much shorter path from the root to the leaf. Normal points, being embedded in dense regions, require many more splits to be isolated. The algorithm averages the path length over an ensemble of such random trees to get a robust anomaly score. This approach avoids computing distances, which become less meaningful and computationally expensive in high-dimensional spaces (the curse of dimensionality), making it very efficient.

The Davies-Bouldin Index aims to find a clustering where each cluster is as compact as possible and as far away from other clusters as possible. The score for each cluster is determined by its worst-case similarity to another cluster. To minimize the overall DBI, we need to minimize the terms for all clusters. The ratio is minimized when the numerator (the sum of intra-cluster scatters) is very small and the denominator (the inter-cluster distance) is very large. This corresponds directly to a configuration of tight, compact clusters whose centers are widely separated. The case where each point is its own cluster (Option B) results in but makes the definition of complex and is generally considered a degenerate case with a poor/undefined score.