Unit 11: Analysis of Variance and Prediction Techniques

I. Foundations of Statistical Analysis

Statistical analysis uses sample data to evaluate differences, assess measurement quality, and predict outcomes. Analysis of variance tests whether group means differ, reliability and validity evaluate measurement instruments, and regression models estimate relationships between dependent and independent variables.

A. Governing Principles and Assumptions

The methods in this unit depend on clearly defined variables, appropriate samples, and models whose assumptions match the data.

• Population and sample: A population contains all units of interest, while a sample of size (n) supplies observations used to estimate population characteristics.
• Parameter and statistic: A parameter, such as population mean (\mu), describes a population; a statistic, such as sample mean (\bar{x}), estimates it.
• Independent and dependent variables: An independent variable represents a predictor, treatment, or grouping factor; a dependent variable is the measured outcome.
• Null hypothesis: Statistical tests ordinarily begin with (H_0), which states that no population difference or relationship exists.
• Significance level: The researcher selects (\alpha), commonly (0.05), as the tolerated probability of rejecting a true null hypothesis.
• Sampling assumptions:
  • Observations should be independently obtained.
  • Samples should represent the intended population.
  • Missing values and influential outliers should be examined.
• Model assumptions: ANOVA and classical regression generally require approximately normal errors and constant error variance; regression also requires an appropriately specified functional relationship.
• Inference and prediction: Inference estimates or tests population relationships, whereas prediction uses an estimated model to forecast an unknown outcome.

II. Analysis of Variance for Mean Difference

A. Analysis of Variance for Mean Difference

Analysis of variance (ANOVA) tests whether the means of three or more populations are equal by comparing variation between groups with variation within groups.

• Hypotheses: For (k) groups, the hypotheses are:

H0: μ1 = μ2 = ... = μk
H1: At least one population mean differs.

Here, (\mu_j) is the population mean of group (j), and (k) is the number of groups.

• Variance partition: Total variation is divided into variation explained by group membership and unexplained variation within groups.

SST = SSB + SSW

Here, (SST) is total sum of squares, (SSB) is between-groups sum of squares, and (SSW) is within-groups sum of squares.

• Sums of squares:

SSB = Σ nj(x̄j - x̄)²
SSW = ΣΣ(xij - x̄j)²

Here, (n_j) is group (j)'s size, (\bar{x}j) is its mean, (\bar{x}) is the grand mean, and (x{ij}) is observation (i) in group (j).

• F statistic: Mean squares standardize the two sources of variation.

MSB = SSB/(k - 1)
MSW = SSW/(N - k)
F = MSB/MSW

Here, (N) is total sample size. A large (F) indicates that observed mean differences are large relative to random within-group variation.

• Decision rule: Reject (H_0) when the p-value is less than (\alpha), or when the calculated (F) exceeds the critical value with (k-1) and (N-k) degrees of freedom.
• Worked example: If (SSB=120), (SSW=180), (k=3), and (N=30), then (MSB=60), (MSW=6.67), and (F\approx9.00). Significance must be determined from the corresponding (F(2,27)) distribution.

B. Applications and Limitations

ANOVA provides an overall test of mean difference, but its interpretation depends on design and assumptions.

• Applications: One-way ANOVA compares groups defined by one factor, such as three teaching methods; factorial ANOVA evaluates two or more factors and their interactions.
• Post-hoc comparisons: A significant omnibus (F) does not identify the differing means; procedures such as Tukey’s HSD compare pairs while controlling familywise error.
• Effect size: Eta squared measures the proportion of total variation associated with group membership.

η² = SSB/SST

Here, (\eta^2) is eta squared; larger values indicate greater explained variation.

• Limitations: Severe non-normality, unequal variances, dependent observations, or influential outliers can distort results. Welch’s ANOVA can address unequal variances, while repeated-measures ANOVA is required for related observations.

III. Reliability of Measurement

A. Reliability

Reliability is the consistency or stability with which an instrument measures a construct across items, occasions, or assessors.

• Classical test theory: An observed score consists of a true component and measurement error.

X = T + E

Here, (X) is the observed score, (T) is the true score, and (E) is random measurement error.

• Test-retest reliability: The same instrument is administered twice to the same respondents; a high correlation between scores indicates temporal stability.
• Parallel-forms reliability: Two equivalent versions of an instrument are administered, and their score correlation measures equivalence.
• Inter-rater reliability: Agreement among observers is evaluated with statistics such as Cohen’s kappa for categorical ratings or the intraclass correlation coefficient for quantitative ratings.
• Internal consistency: Cronbach’s alpha assesses how consistently multiple items measure a common construct.

α = [k/(k - 1)] [1 - (Σsi²/sT²)]

Here, (\alpha) is Cronbach’s alpha, (k) is the number of items, (s_i^2) is item (i)'s variance, and (s_T^2) is total-score variance.

• Interpretation: Alpha ranges theoretically from negative values to (1), although useful scales generally produce positive values. A high value supports consistency but does not prove that all items measure one dimension.

B. Applications and Limitations

Reliability analysis identifies unstable measurements and guides improvement of research instruments.

• Applications: Researchers can standardize instructions, train raters, clarify ambiguous items, and revise poorly functioning questions.
• Error consequences: Low reliability attenuates observed correlations and reduces statistical power because random error obscures systematic differences.
• Limitations: Reliability is sample- and context-dependent. Alpha can rise merely because more similar items are added, and excessive item repetition may narrow construct coverage.
• Necessary distinction: A measure can be reliable without being valid; a consistently miscalibrated scale produces stable but inaccurate readings.

IV. Validity of Measurement

A. Validity

Validity is the degree to which evidence and theory support the interpretation and intended use of scores or research conclusions.

• Face validity: An instrument appears relevant on superficial inspection; this is useful for acceptability but is not strong technical evidence.
• Content validity: Items adequately represent the construct’s full domain. Subject specialists may map test items against a defined content framework.
• Criterion-related validity:
  1. Concurrent validity: Scores agree with a criterion measured at approximately the same time, such as a new anxiety scale compared with an established scale.
  2. Predictive validity: Scores forecast a later criterion, such as an aptitude score predicting subsequent performance.
• Construct validity:
  • Convergent evidence: Scores correlate strongly with theoretically related measures.
  • Discriminant evidence: Scores show weaker relationships with conceptually distinct measures.
• Internal validity: A causal conclusion is credible when alternative explanations such as selection bias, history, maturation, and confounding are controlled.
• External validity: Findings generalize across populations, settings, measurements, and time periods.

B. Applications and Limitations

Validity must be established through accumulated evidence rather than represented by a single universal coefficient.

• Instrument development: Researchers define the construct, prepare representative items, obtain expert review, pilot the instrument, and examine its relation to theoretically relevant variables.
• Dependence on use: A test may support one interpretation but not another; validity for screening does not automatically establish validity for diagnosis.
• Reliability relationship: Reliability limits validity because highly inconsistent scores cannot accurately represent a stable construct, but high reliability alone is insufficient.
• Threats: Restricted samples, inappropriate criteria, response bias, poorly operationalized variables, and cultural differences can weaken validity claims.

V. Bivariate Regression

A. Bivariate Regression

Bivariate, or simple linear, regression models the linear relationship between one predictor (X) and one outcome (Y).

• Population model:

Yi = β0 + β1Xi + εi

Here, (Y_i) is outcome (i), (X_i) is its predictor value, (\beta_0) is the population intercept, (\beta_1) is the population slope, and (\varepsilon_i) is random error.

• Estimated equation:

Ŷ = b0 + b1X
b1 = Σ[(Xi - X̄)(Yi - Ȳ)] / Σ(Xi - X̄)²
b0 = Ȳ - b1X̄

Here, (\hat{Y}) is predicted (Y), (b_0) and (b_1) estimate the intercept and slope, and (\bar{X}) and (\bar{Y}) are sample means.

• Least-squares principle: Coefficients minimize the sum of squared residuals, (\sum(Y_i-\hat{Y}_i)^2).
• Slope interpretation: (b_1) gives the expected change in (Y) for a one-unit increase in (X).
• Coefficient of determination: (R^2) is the proportion of sample variation in (Y) explained by the fitted model; in simple regression, it equals the squared Pearson correlation.
• Worked example: For (\hat{Y}=40+3X), an additional unit of (X) predicts a 3-unit increase in (Y); when (X=5), (\hat{Y}=55).

B. Assumptions and Limitations

Bivariate regression supports estimation and prediction only when its functional form and error assumptions are reasonable.

• Assumptions: The relationship is linear; errors are independent, have mean zero and constant variance, and are approximately normal when tests or intervals rely on normality.
• Diagnostics: Residual plots reveal nonlinearity and heteroscedasticity, while leverage and influence measures identify observations that strongly affect the fitted line.
• Limitations: Correlation and regression do not establish causation. Omitted variables may create a misleading association, and extrapolation beyond observed (X)-values is risky.

VI. Multiple Regression Analysis

A. Multiple Regression Analysis

Multiple regression predicts one quantitative outcome from two or more predictors while estimating each predictor’s association with the outcome conditional on the others.

• Model:

Yi = β0 + β1X1i + β2X2i + ... + βpXpi + εi

Here, (p) is the number of predictors, (X_{ji}) is predictor (j) for observation (i), and (\beta_j) is its partial regression coefficient.

• Coefficient interpretation: (\beta_j) is the expected change in (Y) for a one-unit increase in (X_j), holding all other included predictors constant.
• Categorical predictors: A variable with (g) unordered categories is commonly represented by (g-1) indicator variables relative to a reference category.
• Model fit: (R^2) measures explained variation, while adjusted (R^2) penalizes the inclusion of predictors that provide little improvement.

Adjusted R² = 1 - [(1 - R²)(N - 1)/(N - p - 1)]

Here, (N) is sample size and (p) is the number of predictors.

• Overall test: The regression (F)-test evaluates (H_0:\beta_1=\beta_2=\cdots=\beta_p=0). Individual (t)-tests evaluate whether a particular coefficient differs from zero.
• Prediction: A fitted equation produces a point prediction; a confidence interval estimates a mean response, while a wider prediction interval covers an individual future observation.

B. Applications and Limitations

Multiple regression can improve prediction and statistical control, but coefficient estimates remain dependent on model specification and data quality.

• Applications: It is used to forecast outcomes, adjust for confounders, compare predictor contributions, and test interaction or nonlinear terms.
• Multicollinearity: Strong correlation among predictors inflates coefficient standard errors and makes separate effects unstable; variance inflation factors help diagnose it.
• Overfitting: A model with too many predictors relative to (N) may fit sample noise and perform poorly on new data; validation data or cross-validation assesses generalization.
• Specification risks: Omitted variables, incorrect functional forms, measurement error, influential cases, and endogenous predictors can bias results.
• Interpretive boundary: Holding variables statistically constant does not by itself create experimental control or justify a causal conclusion.