Back to Subject

Unit 5 - Notes

MGN206 6 min read

Unit 5: Correlation and Regression

1. Introduction to Bivariate Analysis

Bivariate analysis involves the simultaneous analysis of two variables (attributes). It explores the concept of association and causality. In research methodology, calculating the strength and direction of the relationship between variables is crucial for hypothesis testing.

  • Correlation: Measures the strength and direction of the association between two variables.
  • Regression: Estimates the nature of the relationship and allows the prediction of one variable (Dependent) based on the value of another (Independent).

2. Correlation Analysis

Correlation determines the degree to which two variables fluctuate together. A positive correlation indicates the extent to which those variables increase or decrease in parallel; a negative correlation indicates the extent to which one variable increases as the other decreases.

Key Concepts

  1. Coefficient of Correlation (): A statistical metric ranging from -1.0 to +1.0.
    • +1.0: Perfect positive correlation.
    • -1.0: Perfect negative correlation.
    • 0: No correlation (independent variables).
  2. Scatter Plot: The graphical representation used to visualize correlation.

Assumptions of Correlation

For the standard correlation coefficient (Pearson) to be valid, specific assumptions must be met regarding the data:

  1. Level of Measurement: Both variables should be continuous (interval or ratio level). For ordinal data, non-parametric equivalents (Spearman) are used.
  2. Related Pairs: Each subject or observation must have both variables (e.g., if measuring height and weight, each data point must come from the same person).
  3. Absence of Outliers: Outliers can skew the correlation coefficient significantly, suggesting a relationship where none exists or hiding one that does.
  4. Linearity: The relationship between the two variables should be linear. If the relationship is curvilinear (e.g., U-shaped), standard correlation analysis will yield misleading results.
  5. Normality: For significance testing of Pearson’s , variables should be approximately normally distributed (bivariate normality).
  6. Homoscedasticity: The variability in scores for variable X should be roughly similar at all values of variable Y.

3. Pearson Product-Moment Correlation ()

The Pearson correlation coefficient is a parametric measure of the linear strength between two variables. It is the most commonly used correlation statistic.

Formula


Where:

  • : Individual data points
  • : Means of the variables

Interpretation Guide

  • 0.00 to 0.30: Negligible correlation
  • 0.30 to 0.50: Low positive/negative correlation
  • 0.50 to 0.70: Moderate positive/negative correlation
  • 0.70 to 0.90: High positive/negative correlation
  • 0.90 to 1.00: Very high positive/negative correlation

Characteristics

  • It is independent of the scale of measurement (unit-less).
  • It is sensitive to outliers.
  • It does not imply causation (Correlation Causation).

4. Spearman’s Rank Correlation ( or )

Spearman’s correlation is a non-parametric test used to measure the degree of association between two variables. It uses ranks rather than raw data.

When to Use

  1. Ordinal Data: When data consists of ranked scores (e.g., Likert scales, class rank).
  2. Non-Normal Distribution: When the assumption of normality required for Pearson is violated.
  3. Monotonic Relationships: When the relationship is not strictly linear but is monotonic (as X increases, Y increases, but not necessarily at a constant rate).
  4. Outliers: Spearman is less sensitive to outliers than Pearson.

Formula

If there are no tied ranks:

Where:

  • : The difference between the two ranks of each observation ().
  • : The number of observations.

(Note: If tied ranks exist, a modified formula utilizing a correction factor is used).

5. Concepts of Regression

While correlation analyzes the strength of association, regression analysis involves identifying the nature of the relationship to perform predictions.

Key Terminology

  1. Independent Variable (IV / Predictor / X): The variable used to predict.
  2. Dependent Variable (DV / Outcome / Y): The variable being predicted.
  3. Deterministic vs. Probabilistic Models:
    • Deterministic: is exactly determined by (e.g., conversion of Celsius to Fahrenheit).
    • Probabilistic: is determined by with some degree of error (used in social sciences/research).

Correlation vs. Regression

Feature Correlation Regression
Objective Measure strength/direction of association. Predict DV based on IV.
Symmetry Symmetric (). Asymmetric (Regression of Y on X X on Y).
Causality No causality implied. Implies a functional relationship (though not necessarily strict causality).
Variables Both represent random variables. IV is fixed/controlled; DV is random.

6. Linear Regression Analysis

Linear regression fits a straight line through the data points that best minimizes the distance between the data points and the line.

Simple Linear Regression Equation

The mathematical model for a straight line is:

  • : The predicted value of the dependent variable.
  • (Alpha): The Y-intercept (the value of when ).
  • (Beta): The Slope (regression coefficient). It represents the change in for every one-unit change in .
  • : The value of the independent variable.
  • (Epsilon): Random error term (residuals).

Method of Least Squares (OLS)

The "Line of Best Fit" is calculated using the Ordinary Least Squares method. This method finds the line where the sum of the squared differences (residuals) between the observed values and the predicted values is minimized.

Coefficient of Determination ()

  • Calculated as the square of the correlation coefficient ().
  • Interpretation: It represents the percentage of variance in the Dependent Variable that is explained by the Independent Variable.
  • Example: If , then . This means 64% of the variation in is explained by , while 36% is due to unexplained factors (error).

Assumptions of Linear Regression

To trust the inferences drawn from a regression model, the LINE assumptions must be met:

  1. L - Linearity: The relationship between X and the mean of Y is linear.
  2. I - Independence of Errors: Observations are independent of each other (no autocorrelation, crucial in time-series data).
  3. N - Normality of Error Distribution: The residuals (errors) should be normally distributed.
  4. E - Equal Variance (Homoscedasticity): The variance of residuals is constant across all levels of X. If the variance changes (funnel shape in residual plot), it is called Heteroscedasticity.

Computational Example (Python Concept)

While manual calculation is possible, research methodology typically utilizes statistical software. Below is a conceptual implementation of Linear Regression.

PYTHON
import numpy as np
from scipy import stats

# Sample Data
x = np.array([1, 2, 3, 4, 5]) # Independent Variable (e.g., Study Hours)
y = np.array([2, 4, 5, 4, 5]) # Dependent Variable (e.g., Test Scores)

# Linear Regression Calculation
slope, intercept, r_value, p_value, std_err = stats.linregress(x, y)

print(f"Equation: Y = {intercept:.2f} + {slope:.2f}X")
print(f"R-squared: {r_value**2:.2f}")

# Interpretation:
# If Slope is 0.6, for every 1 hour of study, the score increases by 0.6 points.

Hypothesis Testing in Regression

  • Null Hypothesis (): (There is no linear relationship; the slope is zero).
  • Alternative Hypothesis (): (There is a significant linear relationship).
  • If the p-value associated with the t-test for the slope is less than the significance level (usually 0.05), we reject the null hypothesis.
Previous
Unit 4
Next
Unit 6