Unit 11: Sampling Theory

I. Foundations of Sampling

Sampling theory provides the mathematical framework for selecting part of a population and using the resulting data to infer characteristics of the whole population.

A. Sampling Theory

Sampling theory connects sample design, probability, estimation, and uncertainty so that conclusions drawn from a sample have measurable reliability.

• Population: The complete collection of units about which information is required; its size is denoted by (N).
• Sample: A subset of (n) population units selected for observation.
• Population parameter: A fixed numerical characteristic, such as population mean (\mu), variance (\sigma^2), or proportion (P).
• Sample statistic: A quantity calculated from sample observations, such as sample mean (\bar X), variance (s^2), or proportion (\hat p).
• Statistical inference: The use of sample statistics to estimate parameters or test claims about them.
• Representative sampling: A good design gives every relevant population group an appropriate probability of selection.
• Sampling error: The random difference between a statistic and its parameter, such as (\bar X-\mu); it arises because only part of the population is observed.
• Nonsampling error: Error caused by nonresponse, faulty measurement, undercoverage, recording mistakes, or biased wording; increasing (n) does not necessarily remove it.
• Core assumptions:
  • Defined population: The target population and sampling units must be specified.
  • Probability mechanism: Selection probabilities must be known for probability-based inference.
  • Independence or adjusted dependence: Observations are either independent or their dependence is included in variance formulas.
  • Adequate sample size: Large-sample approximations require conditions such as finite variance or sufficient expected category counts.

II. Probabilistic Selection — Constructing Unbiased Samples

A. Random Samples and Random Numbers

A random sample is generated by chance rather than investigator choice, while random numbers implement that chance mechanism objectively.

• Simple random sample (SRS): Every possible sample of size (n) has the same probability of selection.
• Independent random sample: Random variables (X_1,\ldots,X_n) are independent and identically distributed from a population with common distribution (F).
• Random-number assignment: Number the (N) units from (1) to (N), generate integers in that range, and select the corresponding units.
• Random-number quality: Software normally produces pseudorandom numbers—deterministic sequences designed to behave statistically like independent uniform observations.
• Seed: A fixed seed reproduces the same pseudorandom sequence, supporting verification of calculations.
• Selection example: For units numbered (1)–(100), five distinct generated integers such as (8,24,51,73,90) identify an SRS of size (5).
• Bias prevention: Convenience selection, voluntary response, and arbitrary substitution violate random selection even when the final sample is large.

III. Replacement Rules — Independence and Finite Populations

A. Sampling with and without replacement

Replacement determines whether a selected unit can appear again and therefore whether successive observations are independent.

1. Sampling with replacement:

   • Procedure: After each draw, return the selected unit before drawing again.
   • Probability: Every draw assigns each of the (N) units probability (1/N).
   • Dependence: Draws are independent, and the same unit can occur more than once.
   • Number of ordered samples: There are (N^n) possible ordered samples of size (n).

2. Sampling without replacement:

   • Procedure: A selected unit cannot be selected again.
   • Changing probability: After one selection, each remaining unit has conditional probability (1/(N-1)).
   • Dependence: Draws are negatively dependent because one unit’s inclusion changes later possibilities.
   • Number of unordered samples:

C(N,n) = N! / [n!(N-n)!]

Here, (N) is population size, (n) is sample size, and (C(N,n)) is the number of distinct samples.

• Finite population correction: Without-replacement sampling reduces variability when the sampling fraction (f=n/N) is appreciable.
• Practical approximation: Dependence is often negligible when (n/N\leq 0.05), so with- and without-replacement formulas become similar.

IV. Distributions of Statistics — Repeated-Sampling Behavior

A. Sampling distributions

A sampling distribution is the probability distribution of a statistic over all possible random samples of a fixed size drawn under the same design.

• Random statistic: Before sampling, a statistic such as (\bar X) varies from sample to sample; after observation, its realized value is fixed.
• Construction: List every possible sample, calculate the statistic for each, and assign probabilities according to the sampling design.
• Center: The expected value indicates whether the statistic is unbiased; (E(\hat\theta)=\theta) means (\hat\theta) is unbiased for (\theta).
• Spread: The variance measures sampling variability; its square root is the standard error.
• Shape: A sampling distribution may be normal, binomial-derived, (t), chi-square, or another form depending on the statistic and assumptions.
• Consistency: A consistent estimator approaches the true parameter in probability as (n) increases.
• Purpose: Confidence intervals, margins of error, and hypothesis tests all depend on sampling distributions rather than merely on the distribution of observed data.

V. Distribution of the Sample Mean

A. Sampling distribution of means

The sampling distribution of (\bar X) describes how sample means fluctuate around the population mean.

• Definition:

X̄ = (X₁ + X₂ + ··· + Xₙ) / n

Here, (X_i) is observation (i), (n) is sample size, and (\bar X) is the sample mean.

• Expected value: For identically distributed observations with mean (\mu),

E(X̄) = μ

Thus, (\bar X) is an unbiased estimator of (\mu).

• Variance with replacement or independent sampling:

Var(X̄) = σ²/n,     SE(X̄) = σ/√n

Here, (\sigma^2) is population variance and (SE(\bar X)) is the standard error.

• Without-replacement variance: For an SRS from a finite population,

Var(X̄) = (1 - n/N)S²/n

Here, (S^2=\sum_{i=1}^{N}(X_i-\mu)^2/(N-1)) is finite-population variance.

• Normal population: If observations are normally distributed, (\bar X) is exactly normal for every (n).
• Central Limit Theorem: For sufficiently large (n), (\bar X) is approximately normal when observations are independent with finite mean and variance:

Z = (X̄ - μ)/(σ/√n) ≈ N(0,1)

• Sample-size effect: Multiplying (n) by four halves (SE(\bar X)); precision improves at the rate (1/\sqrt n), not (1/n).

VI. Categorical Characteristics

A. Sampling distribution of properties

The sampling distribution of properties concerns sample proportions used to measure categorical attributes such as success, defect, or membership.

• Indicator representation: Define (X_i=1) if unit (i) has the property and (X_i=0) otherwise.
• Sample proportion:

p̂ = X/n

Here, (X) is the number possessing the property, (n) is sample size, and (\hat p) is the observed proportion.

• Binomial model: Under independent trials with property probability (P), (X\sim\operatorname{Binomial}(n,P)).
• Mean and variance:

E(p̂) = P
Var(p̂) = P(1-P)/n
SE(p̂) = √[P(1-P)/n]

• Finite-population adjustment: Without replacement, multiply the variance by ((N-n)/(N-1)).
• Normal approximation: The distribution of (\hat p) is approximately normal when (nP) and (n(1-P)) are both sufficiently large, commonly at least (5) or (10).
• Estimated standard error: Because (P) is usually unknown, replace it with (\hat p):

SÊ(p̂) = √[p̂(1-p̂)/n]

VII. Combining Independent and Dependent Statistics

A. Sampling distribution of differences and sums

The distribution of a sum or difference follows from expectation and variance rules, with covariance determining the effect of dependence.

• Expectation rules:

E(X̄₁ ± X̄₂) = μ₁ ± μ₂

Here, (\bar X_1,\bar X_2) are sample means and (\mu_1,\mu_2) are their population means.

• Independent samples:

Var(X̄₁ ± X̄₂) = σ₁²/n₁ + σ₂²/n₂
SE(X̄₁ - X̄₂) = √(σ₁²/n₁ + σ₂²/n₂)

Variances add for both sums and differences because (\operatorname{Var}(-X)=\operatorname{Var}(X)).

• Dependent statistics:

Var(A ± B) = Var(A) + Var(B) ± 2Cov(A,B)

Here, (\operatorname{Cov}(A,B)) measures joint variation; the minus sign applies to (A-B).

• Difference of proportions:

SE(p̂₁-p̂₂) = √[P₁(1-P₁)/n₁ + P₂(1-P₂)/n₂]

This formula assumes independent samples.

• Paired data: For matched observations, calculate differences (D_i=X_i-Y_i) and analyze (\bar D), with (SE(\bar D)=s_D/\sqrt n); treating paired observations as independent discards covariance information.

VIII. Quantifying Sampling Precision

A. Standard errors

A standard error is the standard deviation of a statistic’s sampling distribution and quantifies random sampling uncertainty.

• Interpretation: A smaller standard error means estimates are more tightly concentrated around their expected value.
• Distinction: Standard deviation describes variability among observations; standard error describes variability among statistics across repeated samples.
• Estimated mean error:

SÊ(X̄) = s/√n

Here, (s) is sample standard deviation.

• Standardization: A statistic is converted to a reference distribution by dividing its distance from the hypothesized parameter by its standard error.
• Confidence interval form:

estimate ± critical value × standard error

For a large-sample 95% interval, the normal critical value is approximately (1.96).

• Unknown population variance: For normal data with unknown (\sigma),

T = (X̄-μ)/(s/√n) ∼ t(n-1)

The (t)-distribution has (n-1) degrees of freedom and heavier tails than the standard normal distribution.

• Limitations: Standard-error formulas become misleading under selection bias, unmodeled clustering, severe nonresponse, or incorrect independence assumptions.

IX. Computational Illustration

A. Software demonstration of elementary sampling theory

Software can simulate repeated sampling to make theoretical sampling distributions, bias, and standard errors visible.

• Demonstration goal: Repeatedly sample from a fixed skewed population, calculate (\bar X), and compare the empirical mean and standard deviation with (\mu) and (\sigma/\sqrt n).
• Python simulation:

import numpy as np

rng = np.random.default_rng(2026)
population = rng.exponential(scale=10, size=100_000)

n = 40
repetitions = 20_000
sample_means = np.empty(repetitions)

for r in range(repetitions):
    sample = rng.choice(population, size=n, replace=True)
    sample_means[r] = sample.mean()

mu = population.mean()
sigma = population.std(ddof=0)

print("Population mean:", mu)
print("Mean of sample means:", sample_means.mean())
print("Theoretical SE:", sigma / np.sqrt(n))
print("Simulated SE:", sample_means.std(ddof=1))

• Expected evidence:
  • Unbiasedness: sample_means.mean() should be close to population.mean().
  • Standard error: The simulated standard deviation should be close to sigma / sqrt(n).
  • Central Limit Theorem: Although the parent exponential population is right-skewed, a histogram of 20,000 means for (n=40) should be substantially more bell-shaped.
• Without replacement: Setting replace=False requires the finite-population correction when comparing theoretical and simulated standard errors.
• Reproducibility: default_rng(2026) fixes the seed, so the generated population and simulation can be repeated exactly.
• Simulation limitation: Monte Carlo results approximate theory and contain simulation error; increasing repetitions improves numerical stability but does not correct a flawed sampling design.