Unit 12: Multivariate Analysis

I. Foundations — Studying Several Variables Simultaneously

Multivariate analysis comprises statistical techniques that examine multiple variables or measurements together. Its governing principle is that relationships among variables should be analysed jointly when separate univariate or bivariate analyses would overlook their combined structure.

A. Defining Framework

Multivariate methods describe patterns, test dependence, classify observations, reduce dimensions, and represent complex preferences or similarities.

• Multivariate observation: Each case has measurements on several variables, represented by a vector such as (\mathbf{x}i=(x{i1},x{i2},\ldots,x{ip})), where (p) is the number of variables.
• Data matrix: A dataset with (n) observations and (p) variables is represented by an (n\times p) matrix (\mathbf{X}).
• Interdependence: Methods such as factor analysis and cluster analysis examine relationships without designating a dependent variable.
• Dependence: Methods such as discriminant and conjoint analysis explain or predict one set of variables from another.
• Measurement scale: The choice of technique depends on whether variables are nominal, ordinal, interval, or ratio scaled.
• Common assumptions: Depending on the technique, assumptions may include linearity, adequate sample size, independence, multivariate normality, and absence of extreme outliers.
• Standardisation: Variables measured in different units are commonly converted to (z)-scores:

z_ij = (x_ij - x̄_j) / s_j

Here, (x_{ij}) is observation (i) on variable (j), (\bar{x}_j) is the variable mean, and (s_j) is its standard deviation.

II. Classification — Organising Multivariate Techniques

A. Classification

Classification places multivariate techniques into groups according to their objectives, variable relationships, and measurement requirements.

• Dependence techniques: One or more dependent variables are identified and explained using independent variables.
  • Discriminant analysis predicts membership in predefined groups.
  • Conjoint analysis estimates the contribution of product attributes to preference.
• Interdependence techniques: No variable is formally designated as dependent.
  • Factor analysis identifies latent dimensions among correlated variables.
  • Cluster analysis groups similar observations.
  • Multidimensional scaling maps perceived similarities or dissimilarities.
• Metric-variable methods: Factor analysis normally uses interval- or ratio-scale variables, often through correlations.
• Non-metric methods: Some forms of multidimensional scaling and conjoint analysis can use rankings, categories, or ordered preferences.
• Exploratory methods: Exploratory factor analysis and clustering discover structures not specified in advance.
• Confirmatory or predictive methods: Confirmatory factor analysis tests a hypothesised structure, while discriminant analysis predicts known group membership.
• Selection principle: The research objective comes first; prediction, reduction, grouping, mapping, and preference estimation require different techniques.

III. Important Methods of Factor Analysis — Extracting Latent Dimensions

A. Important Methods of Factor Analysis

Factor analysis explains correlations among observed variables through a smaller number of unobserved factors.

X_j = a_j1F_1 + a_j2F_2 + ... + a_jmF_m + e_j

Here, (Xj) is observed variable (j), (a{jk}) is its loading on factor (k), (F_k) is a common factor, (m<p), and (e_j) is unique error.

1. Principal component analysis: PCA forms components that account for maximum total variance.
   • The first component has the largest eigenvalue; later components explain successively smaller amounts.
   • It uses total variance and is mainly a data-reduction method rather than a strict latent-factor model.
2. Common factor analysis: This method explains shared variance and separates it from unique variance.
   • Principal axis factoring: Estimates communalities and extracts factors from common variance.
   • Maximum likelihood factoring: Estimates parameters by maximising the likelihood of the observed correlation matrix, usually assuming multivariate normality.
   • Image factoring: Predicts each variable from the others and analyses the common part of those predictions.
   • Alpha factoring: Selects factors to maximise the reliability, commonly coefficient alpha, of variables forming each factor.
   • Method choice: PCA suits compression, principal axis factoring suits latent-construct exploration, and maximum likelihood supports statistical tests and confidence intervals.
   • Limitation: Different extraction methods can produce different loadings, especially with small samples, weak communalities, or non-normal data.

IV. Factor Analysis Procedure — From Correlations to Interpretation

A. Factor Analysis Procedure

The factor analysis procedure converts a large correlated variable set into a smaller, interpretable factor structure.

• Problem formulation: Specify the constructs, variables, population, and intended exploratory or confirmatory purpose.
• Data adequacy: Use an adequate sample and inspect missing values, outliers, and correlations.
  • A commonly used guideline is at least five observations per variable, though communalities and factor strength matter more than a fixed ratio.
• Correlation assessment: Variables should show meaningful correlations without near-perfect multicollinearity.
  • Bartlett’s test evaluates whether the correlation matrix differs from an identity matrix.
  • The Kaiser–Meyer–Olkin statistic ranges from 0 to 1; higher values indicate more compact correlations.
• Factor extraction: Choose PCA, principal axis, maximum likelihood, or another justified method.
• Number of factors: Combine theoretical reasoning with eigenvalues, a scree plot, parallel analysis, and interpretability.
  • The eigenvalue-greater-than-one rule retains factors explaining more variance than one standardised variable, but should not be used alone.
• Rotation: Rotate the initial solution to produce a simpler and more interpretable loading pattern.
• Interpretation: Name factors from variables with substantial loadings, while checking cross-loadings and theoretical consistency.
• Factor scores: Estimate each observation’s position on the retained factors for later regression, segmentation, or comparison.
• Validation: Repeat the analysis on another sample or use confirmatory factor analysis to test whether the structure is stable.

V. Rotation in Factor Analysis — Achieving Simple Structure

A. Rotation in Factor Analysis

Rotation changes factor axes to make loadings easier to interpret without changing the basic communalities represented by the retained solution.

1. Orthogonal rotation: Factors remain uncorrelated because the axes stay at (90^\circ).
   • Varimax: Maximises the variance of squared loadings within each factor, encouraging high or low loadings.
   • Quartimax: Simplifies variables by encouraging each variable to load strongly on fewer factors.
   • Equamax: Balances the simplifying objectives of varimax and quartimax.
2. Oblique rotation: Factors may correlate, often making it more realistic for social and behavioural constructs.
   • Direct oblimin: Controls the permitted degree of factor correlation through a parameter.
   • Promax: Begins with an orthogonal solution and transforms it into a correlated structure.
   • Pattern matrix: Under oblique rotation, it contains regression-like coefficients showing each factor’s distinctive contribution.
   • Structure matrix: It reports correlations between variables and factors and may differ from the pattern matrix when factors correlate.
   • Interpretive criterion: A simple structure has strong loadings on one factor and weak loadings elsewhere; for example, ((0.81,0.12)) is clearer than ((0.55,0.49)).
   • Limitation: Rotation improves interpretation but does not rescue unsuitable variables, inadequate samples, or an incorrectly chosen number of factors.

VI. Cluster Analysis — Discovering Natural Groups

A. Overview of Cluster Analysis

Cluster analysis divides observations into internally similar and externally dissimilar groups without using predefined class labels.

• Similarity measurement: Euclidean distance is common for continuous variables:

d(i,k) = √[Σ_j (x_ij - x_kj)²]

Here, (d(i,k)) is the distance between observations (i) and (k), and (j) indexes variables.

• Hierarchical clustering: Builds a nested sequence of clusters displayed in a dendrogram.
  • Agglomerative methods begin with individual cases and merge them.
  • Divisive methods begin with one cluster and repeatedly split it.
• Linkage rules: Single linkage uses nearest members, complete linkage uses farthest members, and Ward’s method minimises increases in within-cluster variance.
• Non-hierarchical clustering: (k)-means assigns observations to (k) centres and iteratively minimises within-cluster squared distances.
• Preprocessing: Standardisation is essential when variables have different units, such as income in rupees and age in years.
• Validation: Examine cluster sizes, within-cluster cohesion, between-cluster separation, stability, and substantive usefulness.
• Applications and limitations: Clustering supports market segmentation and typology development, but results depend heavily on scaling, distance measures, outliers, and the selected number of clusters.

VII. Discriminant Analysis — Predicting Known Group Membership

A. Discriminant Analysis

Discriminant analysis constructs functions of predictor variables that separate predefined categorical groups and classify new observations.

D = b_0 + b_1X_1 + b_2X_2 + ... + b_pX_p

Here, (D) is the discriminant score, (b_0) is a constant, (b_j) is a coefficient, and (X_j) is predictor (j).

• Group structure: The dependent variable is categorical, such as successful versus unsuccessful firms, while predictors are usually metric.
• Objective: Coefficients maximise separation between group centroids relative to variation within groups.
• Functions: With (g) groups and (p) predictors, at most (\min(g-1,p)) discriminant functions can be estimated.
• Assumptions: Classical linear discriminant analysis assumes independent observations, multivariate normal predictors within groups, and equal covariance matrices.
• Interpretation: Standardised coefficients and structure correlations indicate which variables contribute most strongly to discrimination.
• Classification: A case is assigned according to its discriminant scores, group centroids, prior probabilities, and misclassification costs.
• Evaluation: Use a classification matrix, hit rate, sensitivity, specificity, and cross-validation rather than relying on the training accuracy.
• Limitation: Strong non-normality, unequal covariance matrices, multicollinearity, and small groups can weaken estimates; logistic regression may be preferable when assumptions fail.

VIII. Multidimensional Scaling — Mapping Perceived Relationships

A. Multidimensional Scaling

Multidimensional scaling represents similarities or dissimilarities among objects as distances in a low-dimensional perceptual map.

• Input matrix: Data consist of pairwise similarities, dissimilarities, correlations, or distances between (n) objects.
• Spatial principle: Similar objects appear close together, while dissimilar objects appear farther apart.
• Metric MDS: Uses numerical distances and seeks coordinates that reproduce their magnitudes.
• Non-metric MDS: Uses ordinal information and preserves the rank order of dissimilarities rather than exact values.
• Dimensionality: One-, two-, or three-dimensional solutions may be compared, with two dimensions often preferred for visual interpretation.
• Goodness of fit: Stress measures disagreement between observed dissimilarities (\delta{ij}) and fitted distances (d{ij}):

Stress = √[Σ(d_ij - δ_ij)² / Σd_ij²]

Lower stress indicates a closer representation, but dimensional interpretability must also be considered.

• Interpretation: Researchers label dimensions by examining object positions or relating coordinates to external attributes.
• Applications and limitations: MDS produces brand-positioning and preference maps, but configurations may rotate or reflect, dimensions may be subjective, and poor input judgments yield misleading maps.

IX. Conjoint Analysis — Measuring Attribute Trade-offs

A. Conjoint Analysis

Conjoint analysis estimates how respondents value the attributes of a product, service, or policy by analysing evaluations of complete profiles.

• Profile structure: Each alternative combines attribute levels, such as price at ₹500, delivery in two days, and a two-year warranty.
• Part-worth model: Overall utility is decomposed into utilities for attribute levels:

U(X) = β_0 + Σ_j Σ_l β_jl x_jl

Here, (U(X)) is profile utility, (\beta0) is a constant, (\beta{jl}) is the part-worth for level (l) of attribute (j), and (x_{jl}) indicates whether that level is present.

• Data collection: Respondents rank, rate, or choose experimentally designed profiles.
• Traditional conjoint: Ratings or rankings are commonly analysed using regression or monotonic procedures.
• Choice-based conjoint: Respondents select one option from each choice set, and utilities are commonly estimated with logit models.
• Experimental design: Full profiles become impractical when attributes are numerous, so fractional factorial designs select an efficient subset.
• Attribute importance: The utility range of each attribute is divided by the sum of all attribute utility ranges and expressed as a percentage.
• Market simulation: Estimated utilities predict preference shares for proposed combinations, prices, or competing products.
• Applications and limitations: Conjoint analysis supports product design and pricing, but results depend on realistic attributes, manageable tasks, valid designs, and the assumption that stated choices approximate actual behaviour.