Unit 5: Probability

I. Foundations of Probability

Probability is the mathematical study of random experiments: processes whose individual outcomes are uncertain but whose possible outcomes and long-run behavior can be analyzed systematically.

• Sample space: The set of all possible outcomes is denoted by (S); for a die roll, (S={1,2,3,4,5,6}).
• Event: An event (A) is any subset of (S); “rolling an even number” gives (A={2,4,6}).
• Probability measure: A function (P) assigns each event a number satisfying:
  • (0\leq P(A)\leq 1).
  • (P(S)=1).
  • For mutually exclusive events (A_i), (P(\bigcup A_i)=\sum P(A_i)).
• Complement rule: The probability that (A) does not occur is (P(A^c)=1-P(A)).
• Addition rule: For any events (A) and (B),
  (P(A\cup B)=P(A)+P(B)-P(A\cap B)).
• Equally likely outcomes: If (S) contains (n) equally likely outcomes and (A) contains (m), then (P(A)=m/n).
• Discrete and continuous models: Discrete variables take countable values; continuous variables take values over intervals.
• Independence: Events (A) and (B) are independent when (P(A\cap B)=P(A)P(B)).

II. Random Variables — Numerical Outcomes of Experiments

A. random variables

A random variable assigns a real number to every outcome of a random experiment.

• Definition: A random variable is a function (X:S\rightarrow\mathbb{R}), where (S) is the sample space and (\mathbb{R}) is the set of real numbers.
• Discrete variable: A discrete random variable has a finite or countably infinite range, such as the number of heads in five coin tosses.
• Continuous variable: A continuous random variable may assume any value in an interval, such as waiting time measured in seconds.
• Probability mass function: For discrete (X), the PMF is (p(x)=P(X=x)), with (p(x)\geq0) and (\sum_xp(x)=1).
• Probability density function: For continuous (X), a density (f(x)) satisfies:
  
  TEXT

  P(a ≤ X ≤ b) = ∫[a,b] f(x) dx,     ∫[-∞,∞] f(x) dx = 1

  
  Here (a,b) are interval endpoints and (f(x)\geq0).
• Cumulative distribution function: For either type,
  (F(x)=P(X\leq x)). It is non-decreasing, right-continuous, and approaches (1) as (x\rightarrow\infty).

III. Uniform Distribution — Equal Probability Across a Range

A. uniform distribution

A uniform distribution models outcomes that are equally likely over a specified set or interval.

• Discrete form: For (n) possible values (x_1,\ldots,x_n),
  
  TEXT

  P(X = xᵢ) = 1/n

  
  where (n) is the number of equally likely values.
• Continuous form: If (X\sim U(a,b)), its density is:
  
  TEXT

  f(x) = 1/(b-a),  a ≤ x ≤ b;     f(x) = 0 otherwise

  
  Here (a) and (b) are the lower and upper endpoints.
• Parameters: Its mean is ((a+b)/2), while its variance is ((b-a)^2/12).
• Interval probability: For (a\leq c\leq d\leq b),
  (P(c\leq X\leq d)=(d-c)/(b-a)).
• Application: It suits random arrival within a fixed interval only when no subinterval is favored.

IV. Normal Distribution — Symmetric Continuous Variation

A. normal distribution

The normal distribution models continuous measurements concentrated symmetrically around a central value.

• Density: If (X\sim N(\mu,\sigma^2)),
  
  TEXT

  f(x) = [1/(σ√(2π))] exp[-(x-μ)²/(2σ²)]

  
  Here (\mu) is the mean, (\sigma>0) is the standard deviation, and (\sigma^2) is the variance.
• Shape: The curve is bell-shaped and symmetric about (x=\mu); therefore mean, median, and mode all equal (\mu).
• Total area: The density integrates to (1), with half the area on each side of (\mu).
• Standardization: Any normal variable can be converted into (Z\sim N(0,1)):
  
  TEXT

  Z = (X-μ)/σ

• Empirical rule: Approximately (68\%), (95\%), and (99.7\%) of observations lie within one, two, and three standard deviations of (\mu), respectively.
• Limitation: Normal models are inappropriate for strongly skewed or strictly bounded data unless an approximation is justified.

V. Exponential Distribution — Waiting-Time Model

A. exponential distribution

The exponential distribution models the waiting time until the next event in a constant-rate Poisson process.

• Density: If (X\sim\operatorname{Exp}(\lambda)),
  
  TEXT

  f(x) = λe^(-λx),  x ≥ 0

  
  Here (\lambda>0) is the event rate per unit time.
• Distribution function: (F(x)=1-e^{-\lambda x}), so the survival probability is (P(X>x)=e^{-\lambda x}).
• Measures: The mean is (1/\lambda), the variance is (1/\lambda^2), and the standard deviation is (1/\lambda).
• Memoryless property: For (s,t\geq0),
  (P(X>s+t\mid X>s)=P(X>t)); elapsed waiting time does not alter the remaining-time distribution.
• Application: If calls arrive at (\lambda=4) per hour, the mean waiting time is (1/4) hour, or (15) minutes.

VI. Poisson Distribution — Counts in Fixed Intervals

A. Poisson distribution

The Poisson distribution models the number of independent events occurring in a fixed interval at a constant average rate.

• Mass function: If (X\sim\operatorname{Poisson}(\lambda)),
  
  TEXT

  P(X=k) = e^(-λ)λ^k/k!,  k = 0,1,2,...

  
  Here (\lambda>0) is the expected event count and (k) is an observed count.
• Measures: Both the mean and variance equal (\lambda); hence the standard deviation is (\sqrt{\lambda}).
• Assumptions: Events occur independently, the average rate is constant, and simultaneous events are negligible over a very small interval.
• Additivity: If independent (X\sim\operatorname{Poisson}(\lambda_1)) and (Y\sim\operatorname{Poisson}(\lambda_2)), then (X+Y\sim\operatorname{Poisson}(\lambda_1+\lambda_2)).
• Connection: Poisson counts have exponentially distributed interarrival times under the same constant-rate process.

VII. Binomial Distribution — Successes in Repeated Trials

A. binomial distribution

The binomial distribution counts successes in a fixed number of independent Bernoulli trials.

• Conditions: There are (n) trials, each trial has success or failure, trials are independent, and the success probability (p) remains constant.
• Mass function: If (X\sim\operatorname{Bin}(n,p)),
  
  TEXT

  P(X=k) = C(n,k)p^k(1-p)^(n-k)

  
  Here (k=0,\ldots,n), (C(n,k)=n!/[k!(n-k)!]), and (1-p) is the failure probability.
• Measures: The mean is (np), the variance is (np(1-p)), and the standard deviation is (\sqrt{np(1-p)}).
• Example: For four fair coin tosses, (P(X=2)=C(4,2)(1/2)^2(1/2)^2=6/16=0.375).
• Approximation: A Poisson model with (\lambda=np) can approximate the binomial when (n) is large and (p) is small.

VIII. Mean — Expected or Average Value

A. mean

The mean represents the distribution’s numerical center and, for a random variable, its long-run expected value.

• Sample mean: For observations (x_1,\ldots,x_n),
  
  TEXT

  x̄ = (1/n) Σ[i=1 to n] xᵢ

  
  Here (n) is the sample size.
• Discrete expectation: (E[X]=\sum_xx\,p(x)), provided the sum converges.
• Continuous expectation: (E[X]=\int_{-\infty}^{\infty}x f(x)\,dx), provided the integral exists.
• Linearity: For constants (a,b), (E[aX+b]=aE[X]+b).
• Sensitivity: Extreme values strongly affect the arithmetic mean, making it less representative for highly skewed data.

IX. Median — Positional Center

A. median

The median divides an ordered dataset or probability distribution into lower and upper halves.

• Sample rule: After sorting (n) values, the median is the middle value when (n) is odd and the mean of the two middle values when (n) is even.
• Distributional definition: A median (m) satisfies (P(X\leq m)\geq1/2) and (P(X\geq m)\geq1/2).
• Robustness: Unlike the mean, the median changes little when an extreme observation is introduced.
• Interpretation: In a continuous distribution with a unique median, (F(m)=0.5).
• Limitation: The median uses order but not the numerical distances among observations.

X. Mode — Most Frequent Value

A. mode

The mode identifies the value or region with the greatest frequency or probability concentration.

• Discrete data: A mode is a value (x) for which (P(X=x)) or the observed frequency is maximal.
• Continuous data: A mode is a point where the density (f(x)) reaches a maximum; individual point probabilities remain zero.
• Multiplicity: A distribution may be unimodal, bimodal, multimodal, or have no unique mode.
• Advantage: The mode applies to categorical data, such as the most common blood group, for which arithmetic averaging is meaningless.
• Limitation: It can be unstable because small frequency changes may alter the modal value.

XI. Standard Deviation — Measure of Dispersion

A. standard deviation

Standard deviation measures the typical distance of values from their mean in the original measurement units.

• Population variance: For (\mu=E[X]),
  
  TEXT

  Var(X) = E[(X-μ)²] = E[X²]-μ²

• Population standard deviation: (\sigma=\sqrt{\operatorname{Var}(X)}).
• Sample standard deviation:
  
  TEXT

  s = √{Σ[i=1 to n](xᵢ-x̄)²/(n-1)}

  
  Here (n-1) provides the usual unbiased estimator of population variance.
• Interpretation: A small standard deviation indicates concentration near the mean; a large value indicates wider spread.
• Transformation: For constants (a,b), (\operatorname{SD}(aX+b)=|a|\operatorname{SD}(X)); adding (b) does not change spread.

XII. Conditional Probability — Probability Given Information

A. conditional probability

Conditional probability updates the probability of an event using knowledge that another event has occurred.

• Definition: For (P(B)>0),
  
  TEXT

  P(A | B) = P(A ∩ B)/P(B)

  
  Here (P(A\mid B)) is the probability of (A) given (B).
• Multiplication rule: (P(A\cap B)=P(A\mid B)P(B)).
• Independence test: (A) and (B) are independent precisely when (P(A\mid B)=P(A)), assuming (P(B)>0).
• Partition rule: If mutually exclusive events (B_1,\ldots,B_n) cover (S), then:
  
  TEXT

  P(A) = Σ[i=1 to n] P(A | Bᵢ)P(Bᵢ)

• Distinction: Mutually exclusive events cannot occur together, whereas independent events do not influence each other’s probabilities.

XIII. Bayes Theorem — Reversing Conditional Probabilities

A. Bayes theorem

Bayes theorem calculates the probability of a possible cause after observing related evidence.

• Basic form: For (P(B)>0),
  
  TEXT

  P(A | B) = P(B | A)P(A)/P(B)

  
  Here (P(A)) is the prior, (P(B\mid A)) is the likelihood, and (P(A\mid B)) is the posterior.
• Partition form: If (A_1,\ldots,A_n) partition (S),
  
  TEXT

  P(Aⱼ | B) =
  P(B | Aⱼ)P(Aⱼ) /
  Σ[i=1 to n] P(B | Aᵢ)P(Aᵢ)

  
  Here (A_j) is one candidate cause and (B) is observed evidence.
• Interpretation: The denominator is the total probability of observing (B) under all possible causes.
• Example: If disease prevalence is (0.01), sensitivity is (0.90), and false-positive probability is (0.05), then:
  
  TEXT

  P(disease | positive)
  = (0.90)(0.01) / [(0.90)(0.01)+(0.05)(0.99)]
  ≈ 0.154

• Significance: A high likelihood (P(B\mid A)) does not necessarily imply a high posterior (P(A\mid B)); the prior probability can materially affect the result.