Unit 12: Hypothesis Testing

I. Foundations of Hypothesis Testing

Hypothesis testing is an inferential method for using sample evidence to evaluate a claim about a population. It begins with competing statements about an unknown population parameter and applies probability theory to determine whether the observed evidence is sufficiently inconsistent with a baseline claim.

• Governing principle: A sample result may differ from a claimed population value because of random sampling variation; a hypothesis test measures whether the difference is too large to attribute reasonably to chance.
• Population and sample: The population is the complete group of interest, while a sample supplies the observations used in the test—for example, 100 invoices sampled from all invoices issued during a year.
• Parameter and statistic: A parameter, such as the population mean μ or proportion p, is unknown; a statistic, such as the sample mean x̄ or proportion p̂, estimates it.
• Probabilistic decision: A test does not prove a hypothesis with certainty. It quantifies evidence and reaches a decision under a stated probability of error.
• Baseline assumption: Calculations proceed as though the null hypothesis is true, allowing the observed statistic to be compared with its null sampling distribution.
• Significance level: The researcher selects α, commonly 0.05 or 0.01, as the maximum tolerated probability of rejecting a true null hypothesis.
• Evidence standard: Results are usually evaluated through a test statistic, a critical region, or a p-value.
• Decision language: The technically appropriate conclusions are “reject the null hypothesis” or “fail to reject the null hypothesis”; failure to reject is not proof that the null is true.
• Validity conditions: Random or representative sampling, independent observations, correct measurement, and satisfaction of test-specific distributional assumptions support reliable inference.

II. Hypotheses — Competing Claims About a Population

A. Definition of hypothesis

A statistical hypothesis is a precise and testable statement about a population parameter or probability distribution.

• Null hypothesis (H₀): The default statement ordinarily represents no effect, no difference, or an established benchmark. It contains an equality—for example, H₀: μ = 500, where μ is mean daily sales.
• Alternative hypothesis (H₁ or Hₐ): This statement represents the effect, difference, or departure for which evidence is sought.
  1. Two-tailed alternative: H₁: μ ≠ 500 detects departures in either direction.
  2. Right-tailed alternative: H₁: μ > 500 tests specifically for an increase.
  3. Left-tailed alternative: H₁: μ < 500 tests specifically for a decrease.
• Mutual exclusivity: H₀ and H₁ cannot both be true. Together they must cover the possibilities relevant to the research claim.
• Simple hypothesis: A simple hypothesis fully specifies the parameter, as in H₀: p = 0.04, where p is the population defect proportion.
• Composite hypothesis: A composite hypothesis permits multiple parameter values, as in H₁: p > 0.04.
• Research and statistical claims: A managerial claim such as “the campaign increases average spending” must be translated into parameter statements such as H₀: μ_new ≤ μ_old and H₁: μ_new > μ_old.
• Directional choice: The form of H₁ must be fixed before examining results. Selecting a one-tailed test after observing the sample artificially increases the chance of a favorable conclusion.
• Operational precision: A useful hypothesis identifies the population, measurable variable, parameter, comparison value, and direction. “Service improved” is vague; H₁: μ_after < μ_before, where μ measures mean waiting time in minutes, is testable.
• Worked formulation: If a supplier claims that packages contain an average of 1 kilogram, an auditor concerned about underfilling states:

H₀: μ = 1 kg
H₁: μ < 1 kg

Here, μ is the true mean package weight. The left-tailed alternative reflects the specific business concern rather than any departure from 1 kilogram.

III. Testing Procedure — From a Claim to an Evidence-Based Decision

A. Statistical procedure of hypothesis testing

The statistical procedure of hypothesis testing converts a population claim into a controlled decision by comparing sample evidence with what would be expected under H₀.

• Step 1—Define the question: Specify the population, response variable, parameter, benchmark, and practical purpose. A question about mean delivery time concerns μ; a question about the share of late deliveries concerns p.
• Step 2—State the hypotheses: Write H₀ and H₁ before analyzing the data. Equality belongs in H₀, while the research direction determines whether the test is left-tailed, right-tailed, or two-tailed.
• Step 3—Choose α: Set the significance level according to the cost of a false rejection. An α of 0.05 permits a 5% long-run Type I error rate when H₀ is true.
• Step 4—Select the test: Match the method to the parameter, design, sample size, and assumptions.
  • A one-sample z test may assess a mean when the population standard deviation σ is known.
  • A one-sample t test assesses a mean when σ is unknown and estimated by sample standard deviation s.
  • A proportion test assesses a population proportion.
  • Paired tests analyze linked measurements, such as the same stores before and after training.
  • Independent-samples tests compare separate groups, such as customers assigned to two website designs.
• Step 5—Check assumptions: Verify independence, representative sampling, measurement quality, and distributional requirements. For a small-sample t test, strong skewness or extreme outliers can invalidate the model; for large samples, the central limit theorem often makes the sample mean approximately normal.
• Step 6—Calculate the test statistic: Standardize the difference between the observed estimate and its null value.

z = (x̄ - μ₀) / (σ / √n)
t = (x̄ - μ₀) / (s / √n)

Here, x̄ is the sample mean, μ₀ is the mean specified by H₀, σ is the population standard deviation, s is the sample standard deviation, and n is sample size. The denominator is the standard error.

• Step 7—Measure extremeness: Use either of two equivalent decision approaches.
  1. p-value method: The p-value is the probability, assuming H₀, of obtaining a test statistic at least as extreme as the observed one. Reject H₀ when p-value ≤ α.
  2. Critical-value method: Reject H₀ when the test statistic falls in the rejection region. In a two-tailed standard-normal test with α = 0.05, the critical values are approximately −1.96 and 1.96.
• Step 8—State the conclusion: Report the decision in the context of the claim. Statistical significance means that the data are difficult to reconcile with H₀; it does not automatically establish commercial importance or causation.
• Type I error: Rejecting a true H₀ is a false positive, with probability α. For example, a company may adopt a campaign that produces no genuine sales increase.
• Type II error: Failing to reject a false H₀ is a false negative, with probability β. A company may overlook a campaign that genuinely improves sales.
• Power: Test power is the probability of detecting a real effect.

Power = 1 - β

Power generally rises with a larger sample, a larger true effect, lower variability, or a higher α, although raising α also increases Type I error.

• Confidence-interval connection: For a two-tailed test at level α, H₀: μ = μ₀ is rejected exactly when the corresponding (1 − α)100% confidence interval excludes μ₀.
• Worked test: A firm claims mean processing time is 30 minutes. A sample of n = 36 gives x̄ = 28, with known σ = 6. For H₀: μ = 30 against H₁: μ < 30:

z = (28 - 30) / (6 / √36) = -2

The left-tail p-value is approximately 0.0228. At α = 0.05, reject H₀; the sample provides statistically significant evidence that mean processing time is below 30 minutes.

IV. Business Use — Testing Decisions Under Uncertainty

A. Applications of hypothesis testing in business contexts

Businesses use hypothesis tests to distinguish genuine operational or market effects from variation that could plausibly arise by chance.

• Quality control: Manufacturers test whether mean dimensions meet specifications or whether a defect proportion exceeds a threshold. For example, H₀: p ≤ 0.02 versus H₁: p > 0.02 can trigger investigation when defects appear to exceed 2%.
• Marketing experiments: Randomized A/B tests compare conversion rates, click-through rates, or average order values. Testing H₀: p_A = p_B against H₁: p_A ≠ p_B evaluates whether designs perform differently.
• Sales analysis: Firms assess whether promotions change mean revenue per customer. Random assignment supports a causal interpretation; merely comparing customers who chose to use a coupon may introduce selection bias.
• Operations management: Tests evaluate reductions in waiting time, delivery time, inventory shortages, or production downtime. Paired observations are appropriate when the same branches are measured before and after a process change.
• Finance and risk: Analysts may test whether a portfolio’s mean excess return differs from zero or whether default rates have increased. Serial dependence and changing volatility must be considered because ordinary independence assumptions may fail.
• Human-resource decisions: Organizations compare turnover, productivity, or training outcomes across groups. Analyses should use job-relevant measures and account for confounders rather than treating statistical differences as proof of fair or effective policy.
• Customer service: A company may test whether a new support system raises first-contact resolution above a benchmark, such as H₀: p = 0.70 against H₁: p > 0.70.
• Practical significance: A small effect can be statistically significant in a very large sample. Managers should report effect size and confidence intervals—for example, a conversion increase of 0.1 percentage points may not cover implementation costs.
• Multiple testing: Testing many products, segments, or metrics increases false-positive risk. If 20 independent true null hypotheses are each tested at α = 0.05, approximately one false rejection is expected on average.
• Decision costs: The choice of α, target power, and sample size should reflect consequences. A false approval of an unsafe component may justify a lower α, while missing a valuable low-risk interface change may justify prioritizing higher power.
• Ethical and analytical limits: Hypothesis tests cannot repair biased samples, inaccurate records, manipulated stopping rules, or poorly designed studies. Transparent hypotheses, predefined metrics, complete reporting, and relevant business context are essential for defensible decisions.