Unit 5: Correlation and Regression

I. Orientation: Association and Prediction

Correlation and regression are statistical methods for studying relationships between variables. Correlation measures the direction and strength of association, whereas regression develops an equation for estimating one variable from another. Both methods are based on paired observations, such as study hours and examination scores for the same students.

• Variables: A predictor or independent variable is commonly denoted by (X); an outcome or dependent variable is denoted by (Y).
• Paired data: Each observation must contain a meaningful pair ((X_i, Y_i)), where (i) identifies the person or case.
• Association: Correlation describes how variables vary together; it does not establish that one variable causes the other.
• Direction: A positive relationship means larger values of (X) tend to accompany larger values of (Y); a negative relationship means larger values of (X) tend to accompany smaller values of (Y).
• Strength: Correlation coefficients range from (-1) to (+1), with values near either extreme indicating stronger association.
• Prediction: Regression estimates the expected value of (Y) from a specified value of (X), while recognizing that predictions contain error.
• Scale and design: The measurement scale, shape of the relationship, outliers, sample size, and independence of observations determine whether a method is appropriate.

II. Assumption of Correlation: Conditions for Meaningful Association

A. Assumption of Correlation

The assumptions of correlation ensure that a calculated coefficient represents the relationship of interest rather than a distortion caused by unsuitable data.

• Paired observations: Each (X) value must correspond to the correct (Y) value; matching one student’s study hours with another student’s score invalidates the analysis.
• Independence: Observations should generally be independent of one another. Scores from members of the same family, class, or repeated measurements may require special methods.
• Appropriate measurement: Pearson correlation normally requires quantitative interval or ratio variables, such as age in years or income in dollars. Spearman correlation can be used with ordinal ranks.
• Linearity for Pearson’s method: The relationship should be approximately straight-line. A U-shaped relationship can have a Pearson coefficient near zero even when the variables are strongly related.
• Monotonicity for Spearman’s method: As one variable increases, the other should generally move in one direction, although the rate of change need not be constant.
• Outliers: A single unusual pair, such as (X=2) and (Y=95) in an otherwise consistent dataset, can substantially change Pearson’s (r). A scatterplot should be inspected.
• Normality for inference: Testing significance or constructing confidence intervals for Pearson’s correlation commonly assumes bivariate normality, especially in small samples. The coefficient itself can still be computed without perfect normality.
• Causation warning: A correlation of (r=.80) between two variables does not prove that either variable causes the other; a third variable or reverse direction may explain the association.

III. Pearson Correlation: Product-Moment Measurement

A. Pearson Correlation

Pearson correlation measures the strength and direction of a linear relationship between two quantitative variables.

• Definition: Pearson’s coefficient is the standardized covariance of (X) and (Y). It compares how both variables deviate from their means.
• Formula:

r = Σ[(Xᵢ − X̄)(Yᵢ − Ȳ)] /
    √[Σ(Xᵢ − X̄)² × Σ(Yᵢ − Ȳ)²]

• (r) = Pearson correlation coefficient.
• (X_i, Y_i) = values for case (i).
• (\bar X, \bar Y) = sample means.
• (\Sigma) = sum over all paired cases.
  • Interpretation of sign: (r=+.70) indicates a positive linear association; (r=-.70) indicates an equally strong negative linear association.
  • Interpretation of size: (r=0) indicates no linear association, not necessarily no relationship. The practical meaning of values such as .30 or .60 depends on the field and measurement quality.
  • Coefficient of determination: Squaring the coefficient gives (r^2), the proportion of variation linearly shared by the variables.

r² = (r)(r)

• For (r=.60), (r^2=.36), so 36% of the sample variation is linearly associated; the remaining 64% is not explained by this two-variable linear association.
  • Worked example: If examination score and study hours produce (r=.82), students who study more tend to score higher, and the relationship is strong and positive. This result alone does not show that increasing study hours will cause every student’s score to rise.
  • Significance testing: A common null hypothesis is (H_0:\rho=0), where (\rho) is the population correlation. For a sample of size (n), the test statistic is:

t = r√(n − 2) / √(1 − r²)

• (t) = test statistic.
• (r) = observed sample correlation.
• (n) = number of paired observations.
• The statistic is compared with a (t)-distribution having (n-2) degrees of freedom.
  • Limitations: Pearson’s (r) is sensitive to extreme observations, detects linear rather than every possible relationship, and should not be interpreted as evidence of causality.

IV. Spearman Correlation: Rank-Based Association

A. Spearman Correlation

Spearman correlation measures the direction and strength of a monotonic relationship using ranks rather than the raw values.

• Purpose: It is suitable when variables are ordinal, when data are not normally distributed, or when a monotonic relationship is more plausible than a linear one.
• Ranking process: Replace each score with its rank, assigning rank 1 to the smallest value and rank (n) to the largest. Tied values receive average ranks.
• Formula without ties:

ρₛ = 1 − [6Σdᵢ² / n(n² − 1)]

• (\rho_s) = Spearman rank correlation.
• (d_i) = difference between the ranks of (X_i) and (Y_i).
• (n) = number of paired observations.
  • Interpretation: (\rho_s=+1) means identical rank ordering; (\rho_s=-1) means exactly reversed rank ordering; (\rho_s=0) indicates no monotonic association.
  • Worked example: For six students ranked by research skill and ranked by report quality, suppose the sum of squared rank differences is (\Sigma d_i^2=14). Then:

ρₛ = 1 − [6(14) / 6(6² − 1)]
   = 1 − 84/210
   = .60

• The result indicates a moderate positive monotonic association between the two rankings.
  • Ties: The simple formula is derived for data without ties. Statistical software uses a rank-correlation calculation that correctly handles tied ranks.
  • Comparison with Pearson: Pearson uses actual distances between values and measures linear association; Spearman uses order and measures monotonic association. Spearman may be less affected by extreme numerical values, although unusual ranks can still influence the result.
  • Limitations: Rank conversion discards information about exact differences. A Spearman coefficient also does not prove causation and can conceal a relationship that changes direction across the range.

V. Concepts of Regression: Explanation and Prediction

A. Concepts of Regression

Regression is a method for describing the average relationship between an outcome and one or more predictors and for producing predictions from that relationship.

• Regression perspective: Correlation treats (X) and (Y) symmetrically; regression distinguishes a predictor (X) from an outcome (Y). Reversing them generally produces a different regression equation.
• Conditional mean: Regression estimates (E(Y\mid X=x)), meaning the expected or average value of (Y) among cases with predictor value (x).
• Observed and fitted values: (Y_i) is the observed outcome, while (\hat Y_i) is the value predicted by the regression equation.
• Residual: The prediction error for case (i) is:

eᵢ = Yᵢ − Ŷᵢ

• (e_i) = residual.
• (Y_i) = observed outcome.
• (\hat Y_i) = fitted outcome.
  • Least squares principle: The chosen line minimizes the sum of squared residuals:

SSE = Σ(Yᵢ − Ŷᵢ)²

• (SSE) = sum of squared errors or residual sum of squares.
  • Regression to the mean: Extremely high or low observed values tend, on average, to be followed by values closer to the group mean. This statistical tendency is not necessarily evidence of a treatment effect.
  • Prediction limits: Predictions are more reliable within the observed range of (X). Extrapolating beyond that range, such as predicting outcomes for study hours never represented in the sample, is risky.
  • Association versus explanation: A regression equation can predict well without identifying a causal mechanism. Causal interpretation requires an appropriate design, control of confounding, and defensible theoretical reasoning.

VI. Linear Regression Analysis: Fitting and Evaluating a Line

A. Linear Regression Analysis

Linear regression analysis fits a straight-line model to quantify how the expected value of (Y) changes as (X) changes.

• Simple linear model:

Yᵢ = β₀ + β₁Xᵢ + εᵢ

• (Y_i) = observed outcome for case (i).
• (\beta_0) = population intercept.
• (\beta_1) = population slope.
• (X_i) = predictor value.
• (\varepsilon_i) = random error for case (i).
  • Estimated prediction equation:

Ŷ = b₀ + b₁X

• (\hat Y) = predicted outcome.
• (b_0) = estimated intercept.
• (b_1) = estimated slope.
  • Slope interpretation: (b_1) is the predicted change in (Y) for a one-unit increase in (X). If (b_1=4.2), each additional hour of training is associated with an estimated 4.2-unit increase in performance score.
  • Intercept interpretation: (b_0) is the predicted value of (Y) when (X=0). It is meaningful only when (X=0) is plausible and within the relevant context.
  • Coefficient estimates from correlation:

b₁ = r(sᵧ / sₓ)
b₀ = Ȳ − b₁X̄

• (s_x) and (s_y) = sample standard deviations of (X) and (Y).
• (r) = Pearson correlation.
• (\bar X) and (\bar Y) = sample means.
  • Worked example: Suppose (\bar X=10), (\bar Y=70), (r=.60), (s_y=12), and (s_x=3). Then (b_1=.60(12/3)=2.4), and (b_0=70-2.4(10)=46). The fitted model is (\hat Y=46+2.4X); at (X=15), the predicted outcome is (82).
  • Goodness of fit: In simple linear regression, (R^2=r^2). With (r=.60), (R^2=.36), meaning the fitted linear model accounts for 36% of outcome variation in the sample.
  • Regression assumptions: The relationship should be linear; errors should be independent; residual variance should be approximately constant across fitted values; and residuals should be approximately normal when significance tests or confidence intervals are required.
  • Diagnostic evidence: A residual-versus-fitted plot can reveal curvature or unequal spread. A Q-Q plot can assess residual normality, while leverage and influence statistics can identify observations that disproportionately determine the line.
  • Inference for the slope: The null hypothesis (H_0:\beta_1=0) states that the population predictor has no linear effect or association in the model. A confidence interval for (b_1) that excludes zero supports a nonzero linear relationship under the model assumptions.
  • Scope of interpretation: Linear regression provides an estimated conditional relationship and prediction equation. It does not, by itself, establish causality, eliminate omitted-variable bias, or guarantee accurate predictions outside the sampled population and range of data.