Unit2 - Subjective Questions

Eigen decomposition is a way of factoring a square matrix into a set of its eigenvectors and eigenvalues. Specifically, for a square matrix , if there exists a non-zero vector and a scalar such that , then is an eigenvector of and is its corresponding eigenvalue.

Geometric Interpretation:

• Eigenvector (): An eigenvector of a linear transformation is a special vector that, when the transformation is applied, only changes in magnitude, not in direction. It lies along the same span (or opposite direction if ).
• Eigenvalue (): The eigenvalue associated with an eigenvector represents the factor by which the eigenvector is scaled during the linear transformation. If , the vector is stretched; if , it's shrunk; if , its direction is reversed and it's scaled.

In essence, eigenvectors define the "axes" or principal directions along which a linear transformation acts merely as a scaling operation, and eigenvalues quantify the extent of that scaling.

The primary limitations of Eigen decomposition in general machine learning problems stem from the strict requirements on the matrix :

• Square Matrices Only: Eigen decomposition is only defined for square matrices (). Many datasets in machine learning are represented as rectangular matrices (e.g., samples, features, where ). This severely limits its direct applicability.
• Symmetry Requirement for Orthogonal Eigenvectors: For the eigenvectors to form an orthogonal (or orthonormal) basis, which is highly desirable for many applications (e.g., basis transformations), the matrix must be symmetric (). While some important matrices in ML (like covariance matrices) are symmetric, many others are not.
• Existence of Real Eigenvalues/Eigenvectors: Not all square matrices have real eigenvalues and eigenvectors. Some might have complex eigenvalues, which complicates interpretation and use in real-world data analysis where real-valued data is prevalent.
• Non-diagonalizability: A matrix might not be diagonalizable, meaning it might not have a full set of linearly independent eigenvectors. This means it cannot be expressed in the form , where is a diagonal matrix of eigenvalues.

Singular Value Decomposition (SVD) is a powerful matrix factorization technique that decomposes any rectangular matrix (of dimensions ) into three matrices:

Where:

• : An orthogonal matrix whose columns are the left singular vectors of . These vectors form an orthonormal basis for the column space of .
• : An diagonal matrix (or effectively diagonal, containing zeros outside the main diagonal) where the diagonal entries are the singular values of . The singular values are always real and non-negative, and are typically ordered in decreasing magnitude. These values represent the "strength" or "importance" of the corresponding singular vectors.
• : The transpose of an orthogonal matrix . The columns of (or rows of ) are the right singular vectors of . These vectors form an orthonormal basis for the row space of .

SVD provides a stable and general factorization for any matrix, overcoming many limitations of Eigen decomposition.

The geometric intuition behind SVD is that any linear transformation (represented by a matrix ) can be decomposed into a sequence of three fundamental geometric operations:

1. Rotation/Reflection: The matrix rotates or reflects the original basis vectors in the input space (). This aligns the axes of the input space with the directions where the transformation will have its most significant effect.
2. Scaling: The diagonal matrix scales these rotated axes. The singular values () quantify the amount of stretching or shrinking along these new principal directions. Larger singular values correspond to directions of greater variance or importance.
3. Rotation/Reflection: The matrix then rotates or reflects the scaled vectors into their final orientation in the output space ().

Visualization: Imagine a unit sphere in the -dimensional input space. When a matrix transforms this sphere, it deforms it into an ellipsoid in the -dimensional output space. The SVD reveals the principal axes of this ellipsoid. The right singular vectors (columns of ) are the original orthogonal directions in the input space that map to the principal semi-axes of the ellipsoid. The left singular vectors (columns of ) are the directions of these principal semi-axes in the output space. The singular values (diagonal elements of ) are the lengths of these principal semi-axes.

The SVD is intimately related to Eigen decomposition, particularly through the matrices and .

Given the SVD of a matrix :

1. Relating to (for Right Singular Vectors and Singular Values):
   Consider the product :
   
   Since is orthogonal, (identity matrix). Also, is a diagonal matrix whose non-zero entries are .
   
   This equation is an Eigen decomposition of . Therefore:

   • The columns of (right singular vectors of ) are the eigenvectors of .
   • The diagonal entries of are the eigenvalues of . Thus, the singular values of are the square roots of the eigenvalues of : .

2. Relating to (for Left Singular Vectors and Singular Values):
   Consider the product :
   
   Since is orthogonal, . Also, is a diagonal matrix whose non-zero entries are .
   
   This equation is an Eigen decomposition of . Therefore:

   • The columns of (left singular vectors of ) are the eigenvectors of .
   • The diagonal entries of are the eigenvalues of . Thus, the singular values of are also the square roots of the eigenvalues of : .

This shows that SVD can be derived by performing Eigen decomposition on the covariance-like matrices and .

Low-rank approximation using SVD involves constructing a matrix that is an approximation of the original matrix , but with a significantly smaller rank . It is achieved by keeping only the largest singular values and their corresponding left and right singular vectors. Mathematically, if , then , where contains the first columns of , is the diagonal matrix of the top singular values, and contains the first rows of .

Benefits:

• Data Compression: By retaining only a few singular values and vectors, the amount of data required to represent the matrix is drastically reduced. Instead of storing elements, we store elements for , elements for , and elements for . For small , this is a significant saving. This is particularly useful in image compression or storing large datasets.
• Noise Reduction: Real-world data often contains noise, which tends to manifest in directions associated with smaller singular values. By truncating the SVD and discarding these smaller singular values, we effectively remove or significantly reduce the impact of noise, leading to a "cleaner" representation of the underlying data structure. This is because the largest singular values typically capture the most significant, robust patterns in the data, while smaller ones often correspond to minor variations or noise.

The Eckart-Young theorem states that this is the best rank- approximation to in terms of Frobenius norm.

From a geometric perspective, the objective of Principal Component Analysis (PCA) is to find a new set of orthogonal axes (principal components) that capture the maximum variance in the data. Imagine a cloud of data points in a high-dimensional space.

Geometric Objective:

1. First Principal Component: Find the direction (a vector) in the original high-dimensional space along which the data points exhibit the greatest variance. This direction becomes the first principal component.
2. Subsequent Principal Components: Find a second direction, orthogonal to the first, that captures the maximum remaining variance. Continue this process for subsequent components, each orthogonal to the previous ones and capturing the maximal variance in the residual data.

Dimensionality Reduction:

PCA achieves dimensionality reduction by projecting the high-dimensional data onto a lower-dimensional subspace spanned by the top principal components. Since these principal components are chosen to capture the most variance, projecting data onto them minimizes the loss of information (variance) while reducing the number of dimensions. The data points, when projected onto these new axes, are best separated and their intrinsic structure is preserved as much as possible with fewer dimensions. This also means minimizing the reconstruction error when projecting back to the original space.

Let our dataset be , an matrix where is the number of samples and is the number of features. Assume is centered (mean of each feature is zero). Our goal is to find a direction vector (unit vector, ) such that the variance of the projected data along this direction is maximized.

1. Projection: The projection of each data point onto the direction is .

2. Variance of Projected Data: Since the data is centered, the mean of the projected data is 0. The variance of the projected data is given by:
   
   This can be written in matrix form as:
   
   Let be the covariance matrix of the data (since is centered). So, we want to maximize .

3. Optimization Problem: We want to maximize subject to the constraint (because must be a unit vector).
   We can use a Lagrange multiplier to solve this:
   

4. Differentiate and Set to Zero: Take the derivative with respect to and set it to zero:
   
   

5. Interpretation: This is the eigenvalue equation! It means that must be an eigenvector of the covariance matrix , and is its corresponding eigenvalue.

6. Maximizing Variance: To maximize , we substitute :
   
   Since , we have . Therefore, to maximize the variance, we must choose the eigenvector that corresponds to the largest eigenvalue .

Thus, the first principal component is the eigenvector of the covariance matrix with the largest eigenvalue, representing the direction of maximum variance in the data.

PCA and SVD are very closely related, and SVD provides a robust and numerically stable way to compute PCA. In fact, PCA can be seen as a specific application of SVD to the centered data matrix.

Let be an data matrix where is the number of samples and is the number of features, and assume is centered (mean of each column is zero). The covariance matrix is (or for population covariance).

From the PCA derivation, the principal components are the eigenvectors of the covariance matrix .

Now, consider the SVD of the centered data matrix :

From the relationship between SVD and Eigen decomposition, we know:

• The columns of (right singular vectors of ) are the eigenvectors of .
• The squares of the singular values () are proportional to the eigenvalues of (specifically, are the eigenvalues of ).

Therefore:

1. Principal Components: The principal components (the directions of maximum variance) are precisely the right singular vectors (columns of ) of the centered data matrix .
2. Variances Explained: The variance explained by each principal component is proportional to the square of its corresponding singular value (i.e., ).

How SVD is used for PCA:

To perform PCA using SVD:

1. Center the Data: Subtract the mean of each feature from its respective column in the data matrix .
2. Compute SVD: Compute the SVD of the centered data matrix .
3. Extract Principal Components: The columns of are the principal components. They are already sorted by the amount of variance they explain (corresponding to the decreasing singular values in ).
4. Project Data: To reduce dimensionality, select the first columns of (the top principal components), denoted as . The projected data in the new lower-dimensional space is .

SVD is often preferred over directly computing the Eigen decomposition of the covariance matrix because it is more numerically stable, especially for large matrices, and can handle rectangular data matrices directly without explicitly forming (which can be numerically problematic for very wide or very tall matrices).

Choosing the optimal number of principal components () is crucial in PCA, balancing data compression/noise reduction with information retention. Several methods are commonly used:

1. Scree Plot (Elbow Method):

   • Plot the eigenvalues (or percentage of variance explained) in decreasing order against the number of principal components.
   • Look for an "elbow" in the plot, where the rate of decrease of eigenvalues sharply changes. This point suggests that additional components contribute less significant variance and might primarily capture noise.

2. Cumulative Explained Variance:

   • Calculate the cumulative sum of the percentage of variance explained by each principal component.
   • Choose such that a desired percentage of total variance is retained (e.g., 90% or 95%). This is a common and quantitative approach.

3. Kaiser's Criterion:

   • Retain only those principal components whose eigenvalues are greater than 1 (if using the correlation matrix) or greater than the average eigenvalue (if using the covariance matrix with standardized features). The rationale is that if an eigenvalue is less than 1, that component explains less variance than a single original standardized variable.

4. Cross-Validation:

   • Use as a hyperparameter and evaluate the performance of a downstream machine learning model (e.g., classifier, regressor) using cross-validation with varying . Select that yields the best model performance.

5. Domain Knowledge/Interpretability:

   • Sometimes, domain experts might suggest a reasonable number of components based on prior knowledge of the data and what underlying factors are expected. The chosen components should ideally be interpretable if that's a goal.

6. Computational Constraints:

   • In cases of very high-dimensional data, practical computational limits might dictate a smaller , even if more variance could theoretically be captured.

PCA and LDA are both dimensionality reduction techniques, but they operate with different objectives and assumptions:

Principal Component Analysis (PCA):

• Objective: To find directions (principal components) that maximize the total variance in the data. It seeks to capture the most significant data variability regardless of class labels. It is an unsupervised technique.
• Perspective: Looks at the overall structure of the data and aims to project it onto a lower-dimensional subspace where the data points are most spread out.
• What it does: Identifies a new set of orthogonal axes (principal components) that explain the maximum amount of variance in the dataset.
• Input: Requires only the data matrix .
• Use Cases: Data compression, noise reduction, visualization, feature engineering when class labels are unknown or irrelevant to the dimensionality reduction goal.
• Preference: Preferred when the goal is to discover underlying patterns, reduce data complexity, or when class separability is not the primary concern, or when labels are unavailable.

Linear Discriminant Analysis (LDA):

• Objective: To find directions (linear discriminants) that maximize the separability between known classes while minimizing the variance within each class. It is a supervised technique.
• Perspective: Seeks a projection that maximizes the ratio of between-class variance to within-class variance.
• What it does: Identifies a new set of axes that best separate the different classes in the dataset.
• Input: Requires both the data matrix and the class labels .
• Use Cases: Classification preprocessing, feature extraction for classification tasks, face recognition, medical diagnosis.
• Preference: Preferred when the goal is to enhance class separability and the dataset has known class labels. It often leads to better classification performance than PCA if the assumption of normal distribution and equal covariance matrices holds.

Key Differences Summarized:

• Supervised vs. Unsupervised: LDA is supervised (uses class labels), PCA is unsupervised (does not use labels).
• Objective: PCA maximizes total variance; LDA maximizes class separability.
• Max Components: PCA can have up to components; LDA can have at most components, where is the number of classes.
• Assumptions: LDA makes assumptions about data distribution (e.g., Gaussian, equal covariances), while PCA is non-parametric.

In essence, PCA is about finding the best data representation in terms of variance, while LDA is about finding the best data representation for classification.

The core idea behind Linear Discriminant Analysis (LDA) is to find a linear transformation that projects high-dimensional data onto a lower-dimensional space, such that the data points of different classes are well-separated, while data points belonging to the same class are clustered together. Unlike PCA which focuses on maximizing total variance, LDA explicitly considers class labels to find features that are most discriminant.

To achieve this, LDA aims to:

1. Maximize the distance between the means of different classes in the projected space.
2. Minimize the variance within each class in the projected space.

These goals are quantified using scatter matrices:

• Within-Class Scatter Matrix ():

  • measures the scatter (variance) of samples within each individual class. It quantifies how spread out the data points are around their respective class means.
  • A small indicates that samples within each class are tightly clustered.
  • It is calculated by summing the covariance matrices of each class, weighted by the number of samples in that class.
  • For classes, , where , and is the mean of class .

• Between-Class Scatter Matrix ():

  • measures the scatter (variance) of the class means around the overall mean of the entire dataset. It quantifies how well-separated the different class means are from each other.
  • A large indicates that the class means are far apart, implying good class separability.
  • It is calculated based on the difference between each class mean and the global mean of all data points.
  • , where is the number of samples in class , is the mean of class , and is the global mean of all data.

LDA seeks to find a projection matrix that maximizes the ratio . The columns of are the linear discriminants, which are the eigenvectors corresponding to the largest eigenvalues of .

Performing Linear Discriminant Analysis (LDA) typically involves the following steps:

1. Calculate Class Means:

   • For each class (out of classes), calculate the mean vector of the samples belonging to that class. Each is a -dimensional vector if the original data has features.

2. Calculate Global Mean:

   • Calculate the overall mean vector of all samples across all classes in the dataset.

3. Compute Within-Class Scatter Matrix ():

   • For each class , compute its scatter matrix .
   • Sum these individual class scatter matrices to get the total within-class scatter matrix: .

4. Compute Between-Class Scatter Matrix ():

   • For each class , compute the contribution of that class to the between-class scatter: , where is the number of samples in class .
   • Sum these contributions to get the total between-class scatter matrix: .

5. Solve the Generalized Eigenvalue Problem:

   • Find the eigenvectors and eigenvalues that satisfy the equation:
     
   • This is equivalent to solving for the eigenvectors of . Note that must be invertible. If it's singular, techniques like pseudo-inverse or regularization are used.

6. Select Linear Discriminants:

   • Order the eigenvectors (linear discriminants) based on their corresponding eigenvalues in decreasing order. The eigenvectors with the largest eigenvalues are the most discriminant.
   • Select the top eigenvectors (typically , where is the number of classes) to form the projection matrix .

7. Project Data:

   • Transform the original -dimensional data points into the new -dimensional subspace using the projection matrix: .

The projected data can then be used for classification tasks (e.g., with a simple classifier like k-NN or SVM) as it emphasizes class separability.

A significant limitation of Linear Discriminant Analysis (LDA) is its inherent assumption that the covariance matrices of all classes are equal (or very similar). This is often referred to as homoscedasticity.

Why this is problematic in real-world datasets:

• Non-Spherical/Differing Variances: In many real-world datasets, the distribution of features for different classes can vary significantly. Some classes might be tightly clustered (small covariance), while others might be widely spread out (large covariance). Additionally, the shape of the clusters (e.g., spherical vs. elongated) might differ across classes, implying non-equal covariance matrices.
• Suboptimal Projections: If the class covariance matrices are substantially different, LDA's assumption of a shared within-class scatter matrix () might not accurately represent the true within-class variability. Consequently, the projection found by LDA (which optimizes for the ratio of between-class to within-class scatter based on this averaged ) might not be optimal for separating classes with distinct covariance structures.
• Performance Degradation: When the equal covariance assumption is violated, the decision boundaries learned by LDA (which are linear) may not effectively separate the classes, leading to suboptimal classification performance. More complex models like Quadratic Discriminant Analysis (QDA), which allow each class to have its own covariance matrix, would be more appropriate in such scenarios, but they come with a higher risk of overfitting due to more parameters.
• Sensitivity to Outliers: Like many mean-based methods, LDA can be sensitive to outliers, which can distort the class means and covariance estimates, affecting the calculated scatter matrices.

Matrix factorization is a fundamental technique in collaborative filtering for recommendation systems. The core idea is to decompose the sparse user-item interaction matrix into two lower-rank matrices representing latent features for users and items.

High-level Overview:

1. User-Item Interaction Matrix (R): A large, sparse matrix is constructed where rows represent users, columns represent items, and entries represent a user's explicit (e.g., ratings) or implicit (e.g., purchases, views) interaction with an item. Most entries are missing (unknown) because a user interacts with only a small fraction of available items.

2. Factorization into Latent Features: Matrix factorization aims to approximate this matrix by multiplying two much smaller, dense matrices:
   

   • : A user-feature matrix (e.g., , where is the number of users, is the number of latent features). Each row represents user 's preferences across latent features.
   • : An item-feature matrix (e.g., , where is the number of items). Each column represents item 's characteristics across latent features.
     The number of latent features is chosen to be significantly smaller than or .

3. Latent Features: These latent features are not predefined but learned from the data. They can represent abstract concepts like "genre preference," "action-packed vs. romantic," "intellectual vs. casual," or other underlying dimensions that explain user preferences and item attributes.

4. Prediction and Recommendation: Once and are learned, the rating or preference of user for item can be predicted by taking the dot product of their respective latent feature vectors:
   
   For any user , items they haven't interacted with yet are scored, and the top-scoring items are recommended.

5. Learning (Optimization): The matrices and are learned by minimizing a loss function (e.g., squared error) between the predicted ratings and the actual known ratings , usually with regularization terms to prevent overfitting to the sparse known data:
   
   Optimization algorithms like Stochastic Gradient Descent (SGD) or Alternating Least Squares (ALS) are commonly used to solve this.

This approach effectively discovers the underlying factors that drive user preferences and item similarities, leading to highly personalized recommendations.

SVD-based matrix factorization (and its variations) offers several significant advantages for personalized recommendations, primarily due to its ability to uncover latent features:

1. Identification of Latent Features: SVD naturally decomposes the user-item interaction matrix into meaningful, albeit abstract, "latent features" or "factors." These factors represent underlying dimensions that explain why users prefer certain items. For example, in movies, latent features might correspond to genre, actors, director, mood, or target audience. These features are not explicitly defined but are learned from the data.

2. Handling Sparsity: Real-world recommendation matrices are extremely sparse (most users interact with very few items). SVD-based methods excel at generalizing from the limited observed interactions to fill in the missing entries. By reducing the dimensionality, it effectively finds a lower-rank approximation that captures the main patterns, implicitly inferring unknown preferences.

3. Personalization: By representing each user and item as a vector in a shared latent feature space, SVD allows for highly personalized recommendations. A user's preference for an item is estimated by the similarity (e.g., dot product) between their respective latent feature vectors. This directly measures how well an item's latent attributes match a user's latent preferences.

4. Scalability (with modifications): While a naive SVD on a very large, sparse matrix can be computationally expensive, modern matrix factorization techniques inspired by SVD (like Funk SVD, Asymmetric SVD, etc.) use iterative optimization methods (e.g., SGD, ALS) that efficiently learn the user and item latent factor matrices even for massive datasets.

5. Addressing Cold Start (partially): While not a complete solution, latent features can help with the cold start problem to some extent. For new users, recommendations can be made if they rate a few items, as their latent feature vector can be quickly estimated. For new items, if they have some metadata, they can be placed into the latent feature space, allowing existing users to receive recommendations. If completely new and no information is available, it's still challenging.

6. Improved Accuracy: By capturing the latent structure in the data, SVD-based models can often provide more accurate and relevant recommendations compared to simpler methods like neighborhood-based collaborative filtering, especially for diverse preferences.

In essence, latent features act as a powerful intermediate representation that effectively models complex user-item relationships, leading to robust and insightful recommendations.

An orthogonal matrix is a square matrix whose columns (and rows) are orthonormal vectors. This means:

1. Each column vector has a Euclidean norm of 1 ().
2. Any two distinct column vectors are orthogonal (their dot product is 0, for ).

Mathematically, an orthogonal matrix satisfies the property , where is the identity matrix. This implies that .

Significance:

• Eigen decomposition: For a symmetric matrix , its Eigen decomposition is , where is an orthogonal matrix whose columns are the eigenvectors of , and is a diagonal matrix of eigenvalues. The orthogonality of is crucial because it ensures that the eigenvectors form an orthonormal basis, meaning they are mutually independent and span the entire space without redundancy. This simplifies transformations and ensures numerical stability.

• Singular Value Decomposition (SVD): In , both and are orthogonal matrices. Their orthogonality is fundamental because:

  • contains the left singular vectors, forming an orthonormal basis for the column space of . This means the vectors in are perfectly aligned to describe the transformed data's directions.
  • contains the right singular vectors, forming an orthonormal basis for the row space of . These vectors define the principal directions in the original data space.
  • The orthogonality guarantees that rotations and reflections are preserved (no shear or skewing) when applying these transformations, making the decomposition geometrically interpretable as a sequence of rotation, scaling, and another rotation. It also ensures that information is preserved during these transformations and back-transformations, as and . This is vital for applications like low-rank approximation, where we want to accurately reconstruct the original data from its reduced representation.

Eigen decomposition is primarily defined for square matrices. Its factorization requires to be . Many real-world datasets in machine learning are represented by rectangular matrices (e.g., samples and features, where ). If we have a data matrix of size , we cannot directly apply Eigen decomposition to .

Even if we try to work around this by forming the covariance matrix (which is square, ) or (which is square, ), this has limitations:

1. Computational Cost: For very large or , forming or explicitly can be computationally very expensive and memory-intensive, potentially leading to numerical instability due to precision errors.
2. Loss of Information/Focus: While provides information about feature variance and covariance (used in PCA), directly working with itself is often more desirable for general matrix factorization.

How SVD Addresses This Limitation:

Singular Value Decomposition (SVD) is specifically designed to work with any rectangular matrix (of size ). It decomposes into:

• is , is , and is .
• Crucially, SVD does not require the input matrix to be square or symmetric.
• It directly provides an orthonormal basis for both the row space (via ) and the column space (via ) of the original, possibly rectangular, data matrix.

This general applicability makes SVD immensely powerful for dimensionality reduction of any dataset. For instance, in PCA, instead of computing the covariance matrix and then its Eigen decomposition, one can directly compute the SVD of the centered data matrix. The right singular vectors (columns of ) provide the principal components, and the singular values determine the variance explained by each component. This approach is numerically more stable and efficient for large, rectangular data.

Beyond recommendation systems, matrix factorization techniques, particularly SVD, have wide-ranging applications:

1. Image Compression and Denoising:

   • Application: SVD is highly effective for compressing and denoising images. An image can be represented as a matrix (or multiple matrices for color channels). By performing SVD and using a low-rank approximation, one can reconstruct the image using significantly fewer values.
   • How it works: For an image matrix , we compute its SVD . By keeping only the top singular values (and corresponding left and right singular vectors), we get an approximated matrix . The largest singular values typically capture the most significant features and structures of the image, while smaller ones often correspond to noise or fine details. Truncating these smaller singular values leads to a compressed (fewer components needed for storage) and denoised (noise-related components are discarded) version of the image.

2. Topic Modeling (e.g., Latent Semantic Analysis - LSA):

   • Application: In Natural Language Processing (NLP), matrix factorization is used to uncover hidden thematic structures (topics) within a collection of documents. This is the core idea behind Latent Semantic Analysis (LSA).
   • How it works: A document-term matrix is constructed, where rows represent documents and columns represent terms (words), and entries represent term frequency (e.g., TF-IDF). This matrix is often very high-dimensional and sparse. Applying SVD to this matrix () helps in:
     • Dimensionality Reduction: Reducing the number of dimensions from thousands of terms to a few hundred or less latent "topics" (the singular values and vectors).
     • Semantic Grouping: The latent features (topics) captured by SVD can group semantically related words and documents, even if they don't explicitly share common words. For instance, documents about "cars" and "automobiles" might be linked by a common latent topic. This allows for better document retrieval, clustering, and summarization by focusing on the underlying meaning rather than just keywords.

While PCA is a powerful dimensionality reduction technique, it has certain limitations and can provide suboptimal results in specific scenarios:

1. Linearity Assumption: PCA assumes that the principal components are linear combinations of the original features. If the underlying data structure is inherently non-linear (e.g., data lying on a manifold), PCA may fail to capture the true low-dimensional structure. Kernel PCA is an extension that addresses this.
2. Variance Maximization Focus: PCA's objective is solely to maximize variance. It does not consider class labels (unsupervised). If the directions of maximum variance do not align with the directions that best separate different classes, PCA might project data into a lower-dimensional space where classes are less separable, leading to poor performance in classification tasks. In such cases, LDA is often preferred.
3. Sensitivity to Scale: PCA is sensitive to the scaling of the features. Features with larger variances will naturally contribute more to the first principal components. It's often crucial to standardize (scale) features before applying PCA to prevent features with larger numerical ranges from dominating the principal components purely due to their magnitude.
4. Interpretability Issues: While principal components offer directions of maximum variance, their interpretability can be challenging. Each component is a linear combination of all original features, making it difficult to assign a simple meaning to a specific principal component, especially when many features contribute.
5. Outlier Sensitivity: PCA is based on covariance calculations, which are sensitive to outliers. Outliers can significantly skew the calculated principal components, leading to an inaccurate representation of the underlying data structure.
6. Loss of Information (potentially): While PCA aims to minimize information loss in terms of variance, some information might still be lost, especially if the discarded components contain subtle but important signals (e.g., relevant for a specific task but not contributing much to overall variance).