Unit 6: Correlation, Regression and Analysis of Variance

I. Statistical Relationships and Variation

Correlation, regression, and analysis of variance (ANOVA) examine relationships among variables and explain variation in observed data. Correlation measures association, regression models dependence and prediction, while ANOVA compares group means by partitioning total variability.

• Governing principle: Statistical variation can be separated into systematic variation explained by known factors and random variation attributed to uncontrolled causes.
• Variables:
  • Dependent variable: The response or outcome being explained, conventionally denoted by (Y).
  • Independent variable: A predictor or explanatory variable, conventionally denoted by (X).
• Measurement conventions: Pearson correlation and ordinary regression generally require quantitative variables; Spearman correlation operates on ranks; ANOVA compares means of quantitative responses across categorical groups.
• Core assumptions: Depending on the method, observations should be independent, relationships may need to be linear or monotonic, and errors may need normality and constant variance.
• Inferential purpose: Sample statistics such as (r), regression coefficients, and (F)-ratios provide evidence about corresponding population relationships.
• Caution: Association does not establish causation; confounding variables, selection bias, and measurement errors can create misleading patterns.

II. Pearson’s Correlation Coefficient — Linear Association

A. Pearson’s Correlation coefficient

Pearson’s product-moment correlation coefficient measures the strength and direction of the linear relationship between two quantitative variables.

• Definition: For paired observations ((x_i,y_i)), the sample coefficient is

r = Σ[(xᵢ − x̄)(yᵢ − ȳ)] /
    √{Σ(xᵢ − x̄)² Σ(yᵢ − ȳ)²}

Here, (x_i,y_i) are the (i)th observations; (\bar{x},\bar{y}) are sample means; and (r) is the sample correlation.

• Computational form:

r = [nΣxy − (Σx)(Σy)] /
    √{[nΣx² − (Σx)²][nΣy² − (Σy)²]}

Here, (n) is the number of pairs, while each summation extends over all observations.

• Range and meaning:

  • (r=1): perfect positive linear association.
  • (r=-1): perfect negative linear association.
  • (r=0): no linear association, though a nonlinear relationship may exist.
  • Values closer to (\pm1) indicate stronger linear association.

• Properties: The coefficient is dimensionless, symmetric in (X) and (Y), and unaffected by changes of origin or positive scale.

• Testing significance: Under bivariate normality and (H_0:\rho=0),

t = r√(n − 2) / √(1 − r²),    df = n − 2

Here, (\rho) is the population correlation and (df) denotes degrees of freedom.

B. Applications and Limitations

Pearson’s coefficient is most informative when a scatterplot indicates an approximately linear pattern without influential outliers.

• Applications: It quantifies relationships such as height versus weight or expenditure versus income and supports simple regression analysis.
• Assumptions: Independent paired observations, quantitative measurement, linearity, and—when testing—approximate bivariate normality are expected.
• Limitations: Outliers can substantially change (r); restricted ranges weaken it; and a high value neither proves causation nor guarantees accurate prediction.

III. Spearman’s Rank Correlation Coefficient — Monotonic Association

A. Spearman’s Rank correlation coefficient

Spearman’s coefficient measures the strength and direction of a monotonic relationship by replacing observations with their ranks.

• Definition without tied ranks:

rₛ = 1 − [6Σdᵢ² / n(n² − 1)]

Here, (r_s) is Spearman’s coefficient, (d_i) is the difference between the two ranks for pair (i), and (n) is the number of pairs.

• Procedure:

  1. Rank each variable separately, assigning rank 1 consistently to either the smallest or largest value.
  2. Calculate each rank difference (di=R{Xi}-R_{Yi}).
  3. Square and sum the differences, then substitute into the formula.

• Tied observations: Assign each tied value the average of the ranks it occupies. With ties, Spearman’s coefficient is most reliably computed as the Pearson correlation between the two rank variables.

• Interpretation: (r_s=1) represents identical ranking, (r_s=-1) represents exactly reversed ranking, and (r_s\approx0) indicates little monotonic association.

• Distinction from Pearson’s (r): Spearman’s method detects monotonic association, including curved relationships that consistently increase or decrease, whereas Pearson’s method specifically measures linear association.

B. Applications and Limitations

Spearman’s coefficient is appropriate for ordinal data or quantitative data that violate Pearson’s linearity or outlier-sensitivity requirements.

• Applications: It compares judges’ rankings, preference orders, class positions, and monotonic relationships in skewed measurements.
• Advantages: Ranking reduces sensitivity to extreme magnitudes and requires no assumption of bivariate normality.
• Limitations: Converting measurements to ranks discards distance information; many ties reduce discrimination; and non-monotonic associations may still produce a coefficient near zero.

IV. Regression Concepts — Explanation and Prediction

A. Regression Concepts

Regression describes the average dependence of a response variable on one or more predictors and provides an equation for estimation or prediction.

• Simple linear model:

Yᵢ = β₀ + β₁Xᵢ + εᵢ

Here, (Y_i) is response (i), (X_i) is its predictor value, (\beta_0) is the population intercept, (\beta_1) is the population slope, and (\varepsilon_i) is random error.

• Least-squares principle: Estimates (a) and (b) minimize the residual sum of squares:

SSE = Σeᵢ² = Σ[Yᵢ − (a + bXᵢ)]²

Here, (\hat{Y}_i=a+bX_i) is the fitted value and (e_i=Y_i-\hat{Y}_i) is the residual.

• Coefficient interpretation: The slope (b) is the estimated change in mean (Y) for a one-unit increase in (X); the intercept (a) is estimated mean (Y) when (X=0).
• Goodness of fit:

R² = SSR / SST = 1 − SSE / SST

Here, (SST=\sum(Y_i-\bar{Y})^2) is total variation and (SSR=\sum(\hat{Y}_i-\bar{Y})^2) is explained variation.

• Assumptions: Errors are independent, have mean zero and constant variance, and are normally distributed when confidence tests are required.

B. Applications and Limitations

Regression is used for prediction, trend estimation, adjustment for predictors, and assessment of conditional relationships.

• Interpolation: Prediction within the observed (X)-range is generally safer than extrapolation beyond it.
• Diagnostics: Residual plots reveal curvature, unequal variance, unusual observations, and dependence.
• Limitation: An equation may fit observed data well yet fail under extrapolation, omitted-variable bias, structural change, or reverse causality.

V. Regression Lines — Direction-Specific Linear Estimates

A. Regression lines

Two regression lines arise because predicting (Y) from (X) and predicting (X) from (Y) solve different least-squares problems.

1. Regression of (Y) on (X):

Y − Ȳ = bᵧₓ(X − X̄),    bᵧₓ = r(sᵧ/sₓ)

Here, (b_{yx}) is the regression coefficient of (Y) on (X), and (s_x,s_y) are sample standard deviations.

2. Regression of (X) on (Y):

X − X̄ = bₓᵧ(Y − Ȳ),    bₓᵧ = r(sₓ/sᵧ)

Here, (b_{xy}) is the regression coefficient of (X) on (Y).

• Common point: Both lines pass through the centroid ((\bar{X},\bar{Y})).
• Coefficient relationship:

bᵧₓbₓᵧ = r²,    r = ±√(bᵧₓbₓᵧ)

The sign of (r) is the common sign of the two regression coefficients.

• Special cases: If (|r|=1), the lines coincide; if (r=0), they become (Y=\bar{Y}) and (X=\bar{X}), which are perpendicular.

B. Applications and Limitations

The correct regression line must be chosen according to which variable is being predicted.

• Directionality: Use (Y) on (X) to predict (Y), not an algebraic rearrangement of the (X)-on-(Y) equation.
• Regression effect: For imperfect correlation, predicted standardized responses tend to lie closer to their mean than predictor values.
• Limitation: Least-squares lines are sensitive to influential points and do not represent curved patterns adequately.

VI. Multiple Correlation and Regression — Several Predictors

A. Multiple correlation and regression

Multiple regression models one response using two or more predictors, while multiple correlation measures agreement between observed and jointly predicted responses.

• Model:

Yᵢ = β₀ + β₁X₁ᵢ + β₂X₂ᵢ + ··· + βₖXₖᵢ + εᵢ

Here, (X_{ji}) is predictor (j) for observation (i), (\beta_j) is its partial regression coefficient, and (k) is the number of predictors.

• Partial interpretation: (\beta_j) estimates the change in mean (Y) per unit change in (X_j), holding every other included predictor constant.
• Multiple correlation:

R = Corr(Y, Ŷ),    0 ≤ R ≤ 1

Here, (\hat{Y}) is the fitted response and (R) is the multiple correlation coefficient.

• Adjusted fit:

Adjusted R² = 1 − [(1 − R²)(n − 1)/(n − k − 1)]

Here, (n) is sample size; adjustment penalizes predictors that add little explanatory value.

• Overall test: The regression (F)-test examines (H_0:\beta_1=\cdots=\beta_k=0), comparing explained variation per predictor with residual variation per error degree of freedom.
• Multicollinearity: Strong correlation among predictors inflates coefficient variances, making individual effects unstable even when overall (R^2) is high.

B. Applications and Limitations

Multiple regression is valuable when an outcome depends simultaneously on several measurable influences.

• Applications: It supports forecasting, confounder adjustment, and estimation of each predictor’s conditional contribution.
• Diagnostics: Residual plots assess model assumptions, while variance inflation factors identify multicollinearity.
• Limitations: Omitted variables, overfitting, interactions, nonlinear effects, and measurement error can undermine interpretation and prediction.

VII. One-Way ANOVA — Comparing Several Group Means

A. Analysis of Variance: One-way classification

One-way ANOVA tests whether several population means are equal when observations are classified by one categorical factor.

• Hypotheses:

H₀: μ₁ = μ₂ = ··· = μₖ
H₁: At least one population mean differs

Here, (\mu_j) is the mean of group (j), and (k) is the number of groups.

• Variation partition:

SST = SSB + SSE
SSB = Σnⱼ(ȳⱼ − ȳ)²
SSE = ΣΣ(yᵢⱼ − ȳⱼ)²

Here, (n_j) and (\bar{y}j) are group (j)’s size and mean, (\bar{y}) is the grand mean, and (y{ij}) is observation (i) in group (j).

• Test statistic:

F = MSB/MSE
MSB = SSB/(k − 1)
MSE = SSE/(N − k)

Here, (N) is total sample size; under (H_0), (F) has (k-1) and (N-k) degrees of freedom.

• Decision: A sufficiently large (F), equivalently a small (p)-value, indicates evidence that not all means are equal.
• Assumptions: Observations are independent, group errors are approximately normal, and population variances are equal.

B. Applications and Limitations

One-way ANOVA replaces multiple pairwise (t)-tests with a single overall test that controls the initial Type I error rate.

• Applications: It compares mean outcomes across treatments, regions, machines, or teaching methods.
• Follow-up: A significant (F)-test does not identify differing groups; controlled procedures such as Tukey comparisons are then required.
• Limitation: Severe heteroscedasticity, dependence, or influential outliers can invalidate the standard (F)-test.

VIII. Two-Way ANOVA — Variation from Two Factors

A. Analysis of Variance: Two-way classification

Two-way ANOVA studies a quantitative response classified by two categorical factors and can assess their separate and joint effects.

• Model with replication:

Yᵢⱼₗ = μ + αᵢ + βⱼ + (αβ)ᵢⱼ + εᵢⱼₗ

Here, (\mu) is the grand mean; (\alpha_i) and (\betaj) are effects of factor levels (i) and (j); ((\alpha\beta){ij}) is their interaction; (l) indexes replicates; and (\varepsilon_{ijl}) is random error.

• Partition of variation:

SST = SSA + SSB + SSAB + SSE

Here, (SSA) and (SSB) measure main-factor variation, (SSAB) measures interaction variation, and (SSE) measures residual variation.

• Interaction: Interaction exists when the effect of one factor changes across levels of the other; nonparallel mean-profile lines provide a visual indication.
• Tests:

F_A = MS_A/MSE
F_B = MS_B/MSE
F_AB = MS_AB/MSE

Each numerator mean square equals its sum of squares divided by its corresponding degrees of freedom.

• Without replication: With one observation per cell, interaction cannot be estimated separately and is ordinarily absorbed into error; conclusions therefore require an additivity assumption.
• Assumptions: Errors should be independent, normally distributed with mean zero, and have a common variance across factor combinations.

B. Applications and Limitations

Two-way ANOVA evaluates two classification variables efficiently and can remove variation attributable to a blocking factor.

• Applications: It compares treatments across blocks, teaching methods across schools, or machines across operators.
• Interpretive order: When interaction is significant, examine simple effects or cell means before interpreting overall main effects.
• Limitations: Missing cells, unequal sample sizes, heteroscedasticity, and unmodeled interactions complicate sums of squares and interpretation.