Unit 5: Role of Statistics in Data Science

I. Statistical Foundations — From Data to Evidence

Statistics provides methods for collecting, summarizing, analyzing, and interpreting data under uncertainty. In data science, it connects observed patterns to defensible conclusions by distinguishing systematic relationships from random variation.

A. Data science

Data science is an interdisciplinary field that extracts knowledge and supports decisions using data, statistical reasoning, computing, and domain expertise.

• Core components:
  • Statistics: Quantifies variation, estimates unknown quantities, and tests claims.
  • Computing: Enables storage, cleaning, visualization, modeling, and automation at scale.
  • Domain knowledge: Determines whether variables, assumptions, and conclusions are meaningful.
• Statistical roles:
  • Description: Summaries such as the mean, median, standard deviation, and correlation characterize a dataset.
  • Inference: Sample evidence is used to estimate or test claims about a larger population.
  • Prediction: Models estimate unknown or future outcomes while measuring uncertainty.
  • Experimental design: Randomization, controls, and replication help establish causal effects.
• Population and sample:
  • A population is the complete group of interest.
  • A sample is the observed subset used to learn about that population.
  • A parameter, such as population mean (\mu), describes the population; a statistic, such as sample mean (\bar{x}), is calculated from sample data.
• Central principle: Conclusions must account for sampling variability, measurement error, bias, confounding, and model assumptions rather than treating observed patterns as certain truths.
• Typical workflow: Define the question, collect and clean data, explore patterns, formulate hypotheses, select a statistical method, validate assumptions, interpret results, and communicate uncertainty.

II. Statistical Hypothesis Testing — Evaluating Claims with Sample Evidence

Statistical hypothesis testing is a formal decision framework in which observed sample data are compared with what would be expected under a specified claim about a population. Its validity depends on appropriate sampling, suitable model assumptions, and a test chosen before interpreting the results.

A. Hypothesis testing

Hypothesis testing assesses whether sample evidence is sufficiently inconsistent with a baseline population claim.

• Procedure:
  1. State the null and alternative hypotheses.
  2. Select a significance level, commonly (\alpha=0.05).
  3. Choose a test statistic whose distribution under the null hypothesis is known or approximated.
  4. Calculate the test statistic and p-value from the sample.
  5. Reject (H_0) when the evidence meets the decision rule; otherwise, fail to reject (H_0).
• General test statistic:

test statistic = (observed estimate − null value) / standard error

• The observed estimate is a sample statistic.
• The null value is the parameter value asserted by (H_0).
• The standard error measures expected sample-to-sample variation.
  • One-sided test: Detects an effect in a specified direction, such as (\mu>50).
  • Two-sided test: Detects a difference in either direction, such as (\mu\ne50).
  • Interpretive boundary: Failing to reject a hypothesis does not prove it true; it means the available data do not provide sufficiently strong evidence against it.

B. Null hypothesis

The null hypothesis, written (H_0), represents the baseline claim of no effect, no difference, no association, or a specified parameter value.

• Formal role: The sampling distribution of the test statistic is derived under the assumption that (H_0) is true.
• Typical forms:
  • Mean: (H_0:\mu=\mu_0)
  • Difference in means: (H_0:\mu_1-\mu_2=0)
  • Independence: (H_0:) variables (X) and (Y) are independent.
• Equality convention: The null normally contains equality because testing requires a precise reference distribution.
• Decision language:
  1. Reject (H_0): Data are unusually inconsistent with the null model.
  2. Fail to reject (H_0): Evidence is insufficient to reject it; this is not confirmation that the null is correct.
• Example: For a claimed average battery life of 10 hours, the baseline may be (H_0:\mu=10), where (\mu) is the population mean battery life.

C. Alternative hypothesis

The alternative hypothesis, written (H_1) or (H_a), states the effect, difference, or association supported when the null is rejected.

• Forms:
  • Two-sided: (H_a:\mu\ne\mu_0), used when either direction matters.
  • Upper-tailed: (H_a:\mu>\mu_0), used only when an increase is the relevant claim.
  • Lower-tailed: (H_a:\mu<\mu_0), used only when a decrease is relevant.
• Research connection: The alternative usually expresses the substantive research claim, but it must be specified before seeing results to avoid choosing a favorable direction afterward.
• Mutual exclusivity: (H_0) and (H_a) cannot both hold, and together they should cover the parameter possibilities under consideration.
• Example: To investigate whether a redesigned process changes mean completion time from 30 minutes, use (H_0:\mu=30) and (H_a:\mu\ne30).

D. Statistical significance

Statistical significance means that a result meets a predefined threshold for incompatibility with the null hypothesis.

• Significance level: (\alpha) is the maximum long-run probability of rejecting a true (H_0) that the procedure permits, often 0.05.
• Decision rule:

Reject H₀ if p-value ≤ α
Otherwise, fail to reject H₀

• (H_0) is the null hypothesis.
• (\alpha) is the chosen significance level.
  • Practical significance: A statistically significant result may be too small to matter in practice; effect sizes and confidence intervals show magnitude and precision.
  • Sample-size effect: Very large samples can make trivial effects statistically significant, while small samples may miss important effects.
  • Multiple testing: Conducting many tests increases the chance of false positives; adjustments such as Bonferroni control compare each p-value with (\alpha/m), where (m) is the number of tests.
  • Reporting standard: State the test, effect estimate, confidence interval, sample size, p-value, and relevant assumptions rather than reporting “significant” alone.

E. Type 1 and type 2 errors

Type 1 and type 2 errors are the two possible incorrect decisions produced by a hypothesis test.

• Type 1 error: Rejecting (H_0) when (H_0) is true—a false positive.
  • Its probability is (\alpha=P(\text{reject }H_0\mid H_0\text{ true})).
  • Example: Concluding that a model update changes conversion when it actually has no effect.
• Type 2 error: Failing to reject (H_0) when a meaningful alternative is true—a false negative.
  • Its probability is (\beta=P(\text{fail to reject }H_0\mid H_a\text{ true})).
  • Example: Missing a real conversion improvement because the experiment is underpowered.
• Power:

Power = 1 − β

• (\beta) is the Type 2 error probability.
• Power is the probability of detecting a specified effect when it exists.
  • Trade-off: With sample size fixed, lowering (\alpha) generally reduces false positives but increases (\beta); increasing sample size can reduce both risks.
  • Determinants of power: Larger true effects, lower variability, larger samples, and an appropriate test increase power.

F. p-value

A p-value is the probability, assuming (H_0) and the test assumptions are true, of obtaining a test statistic at least as extreme as the observed statistic.

• Conditional meaning:

p-value = P(result at least as extreme as observed | H₀ is true)

• Not represented: It is neither (P(H_0\text{ is true}\mid\text{data})) nor the probability that the result occurred “by chance.”
• Interpretation: A small p-value indicates that the data would be unusual under (H_0); it does not measure effect size or practical importance.
• Threshold use: If (p=0.03) and (\alpha=0.05), reject (H_0); if (\alpha=0.01), fail to reject it.
• Continuous evidence: Values near a threshold should not be treated as fundamentally different—for example, (p=0.049) and (p=0.051) provide nearly identical evidence.
• Dependence on design: Invalid sampling, violated assumptions, selective reporting, or repeated unplanned testing can make a calculated p-value misleading.

III. Comparing Several Means — Variance-Based Inference

Analysis of variance compares group means by separating variability attributable to group membership from unexplained variability within groups. Standard one-way analysis assumes independent observations, approximately normal errors within groups, and equal population variances.

A. ANOVA

ANOVA tests whether three or more population means are all equal without inflating Type 1 error through many separate t-tests.

• Hypotheses:
  • (H_0:\mu_1=\mu_2=\cdots=\mu_k)
  • (H_a:) at least one population mean differs.
  • Here, (\mu_i) is the mean of group (i), and (k) is the number of groups.
• Variance comparison:

F = MS_between / MS_within

• (F) is the ANOVA test statistic.
• (MS_{\text{between}}) measures variation among group means.
• (MS_{\text{within}}) measures variation among observations inside groups.
  • Interpretation: An (F)-ratio near 1 suggests group differences are comparable to ordinary within-group noise; a large (F) provides evidence against (H_0).
  • Degrees of freedom: Between-group degrees of freedom are (k-1); within-group degrees of freedom are (N-k), where (N) is the total sample size.
  • Follow-up analysis: A significant ANOVA shows that some mean differs but not which one; controlled post-hoc comparisons, such as Tukey’s method, identify differing pairs.
  • Limitations: Strong outliers, unequal variances, dependence, or severe non-normality can invalidate conclusions; transformations, Welch’s ANOVA, or nonparametric methods may be preferable.

IV. Categorical Data Analysis — Testing Counts and Associations

Chi-square procedures compare observed category frequencies with frequencies expected under a null model. They apply to counts in mutually exclusive categories and require independent observations with adequately large expected frequencies.

A. Chi-square test

A Chi-square test determines whether categorical counts fit a proposed distribution or whether two categorical variables are associated.

• Test statistic:

χ² = Σ [(Oᵢ − Eᵢ)² / Eᵢ]

• (\chi^2) is the Chi-square statistic.
• (O_i) is the observed count in category or cell (i).
• (E_i) is the expected count under (H_0).
• (\Sigma) means summation across all categories or cells.
  • Goodness-of-fit test: Compares one categorical variable’s observed distribution with specified proportions; degrees of freedom are commonly (k-1).
  • Test of independence: Tests whether two categorical variables in an (r\times c) contingency table are independent.
  • Expected cell count:

Eᵢⱼ = (row i total × column j total) / grand total

• (E_{ij}) is the expected count in row (i), column (j).
• Independence-test degrees of freedom are ((r-1)(c-1)).
  • Decision: A large (\chi^2) and small p-value indicate that observed counts differ more from expected counts than random variation would ordinarily explain.
  • Conditions: Data must be raw counts, categories must not overlap, observations must be independent, and expected counts should generally not be very small.
  • Limitations: Statistical significance does not show the strength or direction of association; standardized residuals identify influential cells, while measures such as Cramér’s (V) quantify association strength.