Unit 6: Correlation, Regression and Analysis of Variance - Subjective Questions

Definition: Pearson’s correlation coefficient measures the strength and direction of the linear relationship between two quantitative variables.

Assumptions:

• Both variables are quantitative.
• The relationship is approximately linear.
• Observations are independent.
• The variables have finite, nonzero variances.
• For inference, approximate bivariate normality and homoscedasticity are commonly assumed.

Properties:

• .
• indicates perfect positive linear correlation, while indicates perfect negative linear correlation.
• indicates no linear correlation, but a nonlinear relationship may still exist.
• It is dimensionless and unaffected by changes of origin and positive scale.
• Correlation does not establish causation.

The means are

Using deviations from the means:

Thus,

Therefore,

Interpretation: There is a moderately strong positive linear relationship between and . Larger values of generally correspond to larger values of .

For variables and , covariance is

Their standard deviations are

Standardizing covariance gives

Substitution and cancellation of produce

By the Cauchy–Schwarz inequality,

Dividing by the positive expression on the right gives . Hence,

Equality occurs when the deviations of one variable are an exact constant multiple of the deviations of the other.

Interpretation:

• The sign indicates direction: is positive association and is negative association.
• The magnitude indicates linear strength: values near or represent strong linear association, while values near represent weak linear association.
• The practical meaning of a given magnitude depends on the subject and data context.

Limitations:

• It measures only linear association.
• It can be strongly influenced by outliers.
• A restricted range of observations can reduce its magnitude.
• Combining distinct groups can produce misleading correlations.
• A high correlation does not prove that one variable causes the other.
• A zero correlation does not imply complete independence, since a strong nonlinear relationship may exist.

Spearman’s rank correlation coefficient measures the strength and direction of a monotonic relationship using ranks rather than original measurements.

When there are no tied ranks,

where is the difference between the paired ranks.

Here,

so

With ,

Interpretation: The two rankings have a strong positive monotonic association.

Tied observations are assigned the average of the ranks they would have occupied. Spearman’s coefficient is then calculated as Pearson’s correlation between the resulting midranks.

The midranks are

Both rank means equal . Therefore,

Hence,

Thus, the variables have a very strong positive monotonic relationship. Calculating Pearson’s correlation of the midranks is preferable to using the simple untied-rank formula directly.

Pearson’s coefficient:

• Uses original quantitative observations.
• Measures linear association.
• Is more sensitive to extreme observations.
• Is appropriate when interval- or ratio-scale data satisfy relevant linear-model assumptions.

Spearman’s coefficient:

• Uses ranks of observations.
• Measures monotonic association, which need not be linear.
• Can be applied to ordinal data.
• Is generally less affected by outliers and skewed distributions.

Common features:

• Both lie between and .
• Their signs show the direction of association.
• Neither coefficient, by itself, establishes causation.

Pearson’s coefficient is preferred for linear quantitative relationships, whereas Spearman’s coefficient is useful for ordinal, non-normal, outlier-prone, or monotonic data.

Regression models the average relationship between a response and one or more predictors and may be used for explanation or prediction.

• Dependent variable: The response whose variation is to be explained or predicted.
• Independent variable: A predictor used to explain variation in .
• Regression coefficient: A parameter describing the expected change in the response for a unit change in a predictor.
• Fitted value: The predicted response obtained from the regression equation.
• Residual: The prediction error
  
• Coefficient of determination: The proportion of sample variation in explained by the fitted model:
  

In simple linear regression with an intercept, , where is Pearson’s correlation coefficient.

Assume the fitted regression line is

The least-squares method minimizes the residual sum of squares

Differentiate with respect to and and set the derivatives equal to zero:

The normal equations are therefore

Solving them gives

Thus, the line can also be written as

Consequently, the least-squares line of on always passes through .

The regression line of on is

where

The regression line of on is

where

Multiplying the coefficients,

Hence,

where the sign of is the common sign of the two regression coefficients.

Key properties:

• Both lines pass through .
• If , the two lines coincide.
• If , the lines become and and are perpendicular.

From the data,

For the regression of on ,

Therefore,

or

For the regression of on ,

Therefore,

or

When , use the regression of on :

Thus, the estimated value of is .

Properties of regression coefficients:

• The two coefficients and have the same sign as .
• Their product is and cannot exceed .
• If one regression coefficient has an absolute value greater than , the other must have an absolute value less than .
• Regression coefficients are unaffected by a change of origin but are affected by a change of scale.

Properties of regression lines:

• Both regression lines pass through .
• The line of on minimizes squared vertical deviations.
• The line of on minimizes squared horizontal deviations.
• The lines coincide under perfect correlation.
• The lines are perpendicular when correlation is zero.

The appropriate line must be chosen according to which variable is being predicted.

A multiple linear regression model with predictors is

Here, is the intercept and is the expected change in for a one-unit increase in , holding all other predictors constant.

Principal assumptions:

• Linearity: The conditional mean of is linear in the model parameters.
• Independence: Error terms are independent.
• Zero conditional mean: .
• Homoscedasticity: Error variance is constant.
• No perfect multicollinearity: No predictor is an exact linear combination of the others.
• Normality for small-sample inference: Errors are normally distributed when exact tests and confidence intervals are required.

Residual plots, influence measures, and multicollinearity diagnostics are used to assess these conditions.

Consider

Least squares minimizes

Setting the partial derivatives of with respect to , , and equal to zero gives

Solving these simultaneous equations produces the least-squares estimates. In matrix notation, the model is

and, when is invertible,

This form generalizes directly to any number of predictors.

Multiple correlation measures the association between and its best linear prediction based jointly on and .

Substitution gives

Therefore,

Partial correlation measures the association between and after removing the linear effect of from both variables:

Thus,

The predictors jointly have a strong association with , while retains a substantial positive association with after controlling for .

One-way ANOVA tests whether the population means of several independent groups are equal under one classification factor.

The model is

where is the overall mean, is the effect of group , and is random error.

Hypotheses:

Assumptions:

• Observations are independent.
• Errors are normally distributed within each population.
• Population variances are equal.
• The response is quantitative.

The test statistic is

A sufficiently large value, or a -value below the significance level, leads to rejection of . Rejection establishes that not all means are equal, but follow-up comparisons are needed to identify which means differ.

The group means and grand mean are

The between-group sum of squares is

The within-group sums of squares are , , and , so

Therefore,

The ANOVA table is:

Thus,

This large value provides strong evidence against equality of the three population means when compared with the relevant critical value. Hence, the null hypothesis would be rejected at conventional significance levels.

Let be observation in group , the mean of group , and the grand mean. Write

Squaring and summing gives

For every group,

so the cross-product term vanishes. Therefore,

where

The corresponding degrees of freedom also partition as

Two-way ANOVA studies the effects of two classification factors, say and , on a quantitative response.

With replication, its model is

where and are main effects and is the interaction effect.

Hypotheses include:

• Factor : all .
• Factor : all .
• Interaction: all .

Interaction means that the effect of one factor changes across levels of the other factor. When interaction is significant, main effects must be interpreted carefully.

Assumptions:

• Independent observations.
• Normally distributed errors.
• Equal error variances across cells.
• Correct additive or interaction model.

With one observation per cell, the additive model

is commonly used, but interaction cannot be estimated separately from error without replication.

There are rows, columns, and observations. The row totals are , , and ; the column totals are , , and . The grand total is .

The correction factor is

Since the sum of squared observations is ,

The row sum of squares is

The column sum of squares is

The residual sum of squares is

The ANOVA table is:

Thus, the row statistic is and the column statistic is . Each must be compared with the relevant critical value. Because there is no replication, the residual term also contains any unmodeled interaction.