Unit 13: Tests of Significance

I. Foundations of Significance Testing

A test of significance evaluates whether sample evidence is sufficiently inconsistent with a null hypothesis (H_0). A test statistic summarizes the discrepancy between observed data and (H_0), while its sampling distribution determines the probability of obtaining such evidence by chance.

Defining framework:

• Null and alternative hypotheses: (H_0) states the value or equality being tested; (H_1) states the competing one-sided or two-sided claim.
• Significance level: (\alpha), commonly (0.05) or (0.01), is the maximum chosen probability of rejecting a true (H_0).
• Critical region: Values beyond distributional critical values lead to rejection of (H_0).
• p-value: The probability, assuming (H_0), of obtaining a statistic at least as extreme as that observed.
• Errors:
  • Type I error: Rejecting a true (H_0), with probability (\alpha).
  • Type II error: Failing to reject a false (H_0), with probability (\beta).
• Decision convention: Reject (H_0) when (p\leq\alpha); otherwise, fail to reject it. Failure to reject does not prove (H_0).

A. Tests based on t, F and Z distributions

Tests based on (t), (F), and (Z) distributions standardize sample evidence according to the parameter, available information, and sampling assumptions.

• Z test: Used chiefly for means when the population standard deviation (\sigma) is known or the sample is large, and for large-sample proportions.

Z = (x̄ − μ₀)/(σ/√n)

Here, (\bar{x}) is the sample mean, (\mu_0) is the hypothesized population mean, (\sigma) is the population standard deviation, and (n) is sample size.

• t test: Used for means when (\sigma) is unknown and is estimated by the sample standard deviation (s); its degrees of freedom reflect estimation uncertainty.
• F test: Uses a ratio of independent variance estimates and is central to variance comparisons and analysis of variance.
• Tail selection:
  1. One-tailed test: (H_1) specifies a direction, such as (\mu>\mu_0); all (\alpha) lies in one tail.
  2. Two-tailed test: (H_1) states inequality, such as (\mu\neq\mu_0); (\alpha/2) lies in each tail.
• Common assumptions: Random sampling, independence, an appropriate population model, and a statistic whose stated sampling distribution applies under (H_0).

II. Student’s t-Distribution — Inference with an Estimated Standard Deviation

The (t)-distribution, developed from William Sealy Gosset’s work published under “Student” (1908), permits inference about means when population variability is unknown.

A. Student’s t-distribution

Student’s (t)-distribution is the sampling distribution of a standardized normal mean when its standard error is estimated from the same sample.

• Construction: If (Z\sim N(0,1)), (U\sim\chi^2_\nu), and (Z) and (U) are independent, then:

T = Z/√(U/ν)

Here, (T) is the t-variable, (U) is chi-square distributed, and (\nu) is the number of degrees of freedom.

• Sample form: For a random sample from (N(\mu,\sigma^2)):

T = (x̄ − μ)/(s/√n),     ν = n − 1

Here, (\mu) is the population mean and (s) is the sample standard deviation.

• Purpose: Replacing unknown (\sigma) by (s) introduces extra uncertainty, represented by heavier tails than those of the standard normal distribution.

B. Definition and properties of t-distribution

The t-distribution is a continuous, symmetric family indexed by its degrees of freedom.

• Density:

f(t) = Γ[(ν+1)/2] / {√(νπ) Γ(ν/2)} · (1+t²/ν)^[−(ν+1)/2]

Here, (t) is a real value, (\nu>0), (\Gamma) is the gamma function, and (f(t)) is the probability density.

• Symmetry: The distribution is symmetric about zero, so (P(T<-a)=P(T>a)).
• Centre: Its mean is (0) for (\nu>1); its mode and median are always (0).
• Spread: Its variance is (\nu/(\nu-2)) for (\nu>2), exceeding the standard normal variance of (1).
• Tail behavior: Small (\nu) produces heavier tails because (s) estimates (\sigma) imprecisely.
• Limiting form: As (\nu\to\infty), the t-distribution approaches (N(0,1)).

C. Critical value of t

A critical value of (t) marks the boundary between the non-rejection and rejection regions for specified degrees of freedom and significance level.

• Notation: (t{\alpha,\nu}) commonly denotes the value satisfying (P(T>t{\alpha,\nu})=\alpha).
• One-tailed rule: Reject an upper-tailed (H0) when (t{\text{obs}}>t_{\alpha,\nu}).
• Two-tailed rule: Reject when:

|t_obs| > t_(α/2,ν)

Here, (t_{\text{obs}}) is the observed statistic.

• Concrete values: For (\nu=10), the upper-tail critical value at (\alpha=0.05) is approximately (1.812), while the two-tailed (5\%) boundary is approximately (\pm2.228).
• Interpretation: Lower degrees of freedom require larger absolute critical values; therefore stronger sample evidence is needed.

D. Applications of t-distribution

The t-distribution supports small-sample inference involving means and related statistics when normal-model conditions are reasonable.

• Mean inference: It provides one-sample confidence intervals and hypothesis tests when (\sigma) is unknown.
• Two-sample inference: It compares independent population means using pooled or Welch standard errors.
• Paired observations: Before-and-after or matched measurements are reduced to one sample of differences.
• Regression and correlation: Individual regression coefficients and a Pearson correlation can be tested using t-statistics.
• Limitations: Strong skewness and extreme outliers can invalidate small-sample t procedures; randomization and independence remain essential.
• Robustness: With moderate or large samples, t procedures often tolerate modest non-normality because the sample mean becomes approximately normal.

III. t-Tests for Population Means

A t-test compares an observed mean difference with its estimated standard error, converting the evidence into a statistic measured on a t scale.

A. Test for a single mean

The one-sample t-test examines whether the mean (\mu) of one population equals a specified value (\mu_0).

• Hypotheses: A two-sided test uses (H_0:\mu=\mu_0) against (H_1:\mu\neq\mu_0).
• Statistic:

t = (x̄ − μ₀)/(s/√n),     ν = n − 1

• Conditions: Observations should form an independent random sample; for small (n), the population should be approximately normal without severe outliers.
• Worked example: If (n=16), (\bar{x}=52), (s=8), and (\mu_0=48), then:

t = (52 − 48)/(8/√16) = 2.00,     ν = 15

At a two-sided (5\%) level, the critical values are approximately (\pm2.131). Since (2.00<2.131), (H_0) is not rejected.

• Confidence-interval link: (H_0) is rejected in a two-sided level-(\alpha) test exactly when (\mu_0) lies outside the corresponding (100(1-\alpha)\%) confidence interval.

B. t-test for difference of means

The two-sample t-test determines whether two independent population means differ significantly.

1. Pooled-variance test: When population variances can reasonably be assumed equal, use:

t = (x̄₁ − x̄₂ − Δ₀) / {s_p√(1/n₁ + 1/n₂)}

s_p² = [(n₁−1)s₁² + (n₂−1)s₂²]/(n₁+n₂−2)
ν = n₁+n₂−2

2. Welch test: When equal variances are not assumed, use:

t = (x̄₁ − x̄₂ − Δ₀)/√(s₁²/n₁ + s₂²/n₂)

• Symbols: (\bar{x}_i), (s_i^2), and (n_i) are group (i)’s mean, variance, and size; (\Delta_0) is the hypothesized mean difference; (s_p^2) is the pooled variance.
• Welch degrees of freedom:

ν ≈ (s₁²/n₁ + s₂²/n₂)²
     / [(s₁²/n₁)²/(n₁−1) + (s₂²/n₂)²/(n₂−1)]

• Choice: Welch’s test is generally safer when sample sizes or variances differ substantially.
• Distinction: Matched observations are not independent samples; they require a one-sample t-test on paired differences.

IV. Fisher’s Transformation of Correlation

Correlation coefficients have a bounded, skewed sampling distribution, especially when the population correlation is far from zero. Fisher’s transformation converts Pearson’s (r) to an approximately normal variable.

A. Fisher Z-transformation

Fisher Z-transformation enables confidence intervals and significance tests for population correlations.

• Transformation:

z_r = ½ ln[(1+r)/(1−r)] = arctanh(r)

Here, (r) is the sample Pearson correlation, (z_r) is its transformed value, and (\ln) is the natural logarithm.

• Sampling model: For a bivariate normal population and sufficiently large (n):

z_r ≈ N(arctanh(ρ), 1/(n−3))
SE(z_r) = 1/√(n−3)

Here, (\rho) is the population correlation and (SE) denotes standard error.

• Confidence interval: Form (zr\pm z{\alpha/2}/\sqrt{n-3}), then back-transform each endpoint using:

r = (e^(2z) − 1)/(e^(2z) + 1) = tanh(z)

• Comparing independent correlations:

Z = (z_r1 − z_r2)/√[1/(n₁−3) + 1/(n₂−3)]

• Limitation: This formula does not directly compare overlapping or dependent correlations; their covariance must be incorporated.

V. F-Distribution and Variance-Ratio Testing

The F-distribution describes the ratio of two independent estimates of variance. It is indexed by separate numerator and denominator degrees of freedom and takes only non-negative values.

A. F-statistic

An F-statistic measures how large one variance estimate is relative to another.

• Definition: If (U1\sim\chi^2{\nu_1}) and (U2\sim\chi^2{\nu_2}) independently, then:

F = (U₁/ν₁)/(U₂/ν₂) ∼ F(ν₁,ν₂)

Here, (\nu_1) and (\nu_2) are numerator and denominator degrees of freedom.

• Variance test: Under (H_0:\sigma_1^2=\sigma_2^2):

F = s₁²/s₂²,     ν₁ = n₁−1,     ν₂ = n₂−1

• Shape: The F-distribution is right-skewed, begins at zero, and becomes less skewed as both degrees of freedom increase.
• Reciprocal property: If (F\sim F(\nu_1,\nu_2)), then (1/F\sim F(\nu_2,\nu_1)).
• Sensitivity: Classical variance-ratio tests are highly sensitive to departures from population normality.

B. Critical value of F-distribution

A critical value of F is an upper-tail boundary determined by (\alpha) and both degrees of freedom.

• Notation:

P[F > F_(α;ν₁,ν₂)] = α

• Upper-tail test: Reject (H0) when the calculated ratio exceeds (F{\alpha;\nu_1,\nu_2}).
• Two-sided variance test: Place (\alpha/2) in each tail or arrange the larger sample variance in the numerator and use the corresponding adjusted upper-tail procedure.
• Asymmetry: Interchanging (\nu_1) and (\nu_2) generally changes the critical value.
• Table use: Select the column for numerator degrees of freedom, the row for denominator degrees of freedom, and the required upper-tail probability.

C. Applications of F-distribution

The F-distribution is used wherever independent estimates partition or compare variability.

• Equality of variances: The two-variance F-test evaluates (H_0:\sigma_1^2=\sigma_2^2) for independent normal populations.
• Analysis of variance: One-way ANOVA uses:

F = MS_between/MS_within

Here, (MS{\text{between}}) measures variation among group means and (MS{\text{within}}) measures variation within groups.

• Regression significance: The overall F-test compares variation explained by a regression model with unexplained residual variation.
• Nested models: A partial F-test determines whether additional predictors produce a significant reduction in residual sum of squares.
• Relationship to t: For a single restriction, an F-statistic with one numerator degree of freedom equals the square of the corresponding t-statistic:

F(1,ν) = t(ν)²

• Interpretive caution: A significant F-test identifies evidence of unequal variation or model effects, but follow-up analysis is required to locate specific group differences.