Unit 1: Introduction to Probability

I. Orientation — The Mathematical Framework of Uncertainty

Probability is the mathematical study of uncertain outcomes. Its modern axiomatic foundation, formulated by Andrey Kolmogorov (1933), represents events as sets and assigns them numerical probabilities satisfying fixed rules. These ideas support statistical inference, risk assessment, reliability analysis, and random-process modelling.

• Experiment: A repeatable process with an uncertain result, such as rolling a die or measuring a component’s lifetime.
• Outcome: One possible result of an experiment; for a die roll, an outcome is a number from 1 to 6.
• Sample space: The set (S) of every possible outcome.
• Event: A subset (A\subseteq S); it occurs when the observed outcome belongs to (A).
• Probability convention: Every event receives a value between (0) and (1), where (0) represents impossibility and (1) represents certainty.
• Modelling assumption: Outcomes are not automatically equally likely; equal probabilities must follow from the experiment’s design or symmetry.
• Core relationship: Set operations describe combinations of events, while probability measures quantify their likelihood.

II. Elements of Set Theory — The Language of Events

A. Elements of Set Theory

Set theory provides the notation and operations needed to define and combine events.

• Membership and containment: (x\in A) means outcome (x) belongs to event (A); (A\subseteq B) means every outcome in (A) also lies in (B).
• Union: (A\cup B) occurs when (A), (B), or both occur.
• Intersection: (A\cap B) occurs only when both events occur.
• Complement: (A^c=S\setminus A) occurs when (A) does not occur.
• Difference: (A\setminus B=A\cap B^c) contains outcomes in (A) but not in (B).
• Special sets: The empty set (\varnothing) is the impossible event, while (S) is the certain event.
• Disjoint events: (A) and (B) are mutually exclusive when (A\cap B=\varnothing).
• De Morgan’s laws: These convert complements of unions into intersections and conversely.

(A ∪ B)ᶜ = Aᶜ ∩ Bᶜ
(A ∩ B)ᶜ = Aᶜ ∪ Bᶜ

Here, (\cup) denotes union, (\cap) intersection, and the superscript (c) complement.

• Example: For (S={1,2,3,4,5,6}), let (A={2,4,6}) and (B={4,5,6}). Then (A\cap B={4,6}), (A\cup B={2,4,5,6}), and (A^c={1,3,5}).

III. Probability Model — Outcomes and Their Numerical Weights

A. Sample Space and Probability Measure

A probability model consists of a sample space, a collection of admissible events, and a probability measure satisfying Kolmogorov’s axioms.

• Sample-space forms:
  • Finite: (S={H,T}) for a coin toss.
  • Countably infinite: (S={1,2,3,\ldots}), such as the trial number of the first success.
  • Continuous: (S=[0,\infty)), such as a nonnegative waiting time.
• Probability measure: (P(A)) assigns a numerical probability to each event (A).
• Kolmogorov’s axioms:

P(A) ≥ 0
P(S) = 1
P(⋃ᵢ Aᵢ) = Σᵢ P(Aᵢ), if Aᵢ ∩ Aⱼ = ∅ for i ≠ j

Here, (P) is the probability measure; (S) is the sample space; (A_i) and (A_j) are events; and (i,j) are indices.

• Derived rules:

P(∅) = 0
P(Aᶜ) = 1 − P(A)
P(A ∪ B) = P(A) + P(B) − P(A ∩ B)

Here, (A) and (B) are events, and the subtraction prevents outcomes in their intersection from being counted twice.

• Equally likely finite outcomes: If (S) has (n) equally likely outcomes and (A) contains (m) of them, then (P(A)=m/n).

IV. Independence — Absence of Probabilistic Influence

A. Statistical Independence

Statistical independence means that knowing whether one event occurred does not alter the probability assigned to another.

• Formal condition:

P(A ∩ B) = P(A)P(B)

Here, (A) and (B) are events, (A\cap B) is their joint occurrence, and (P(A)P(B)) is the product of their individual probabilities.

• Equivalent conditional form: If (P(B)>0), independence implies (P(A\mid B)=P(A)).
• Symmetry: Independence is reciprocal; if (A) is independent of (B), then (B) is independent of (A).
• Complements: If (A) and (B) are independent, then (A^c) and (B), (A) and (B^c), and (A^c) and (B^c) are also independent.
• Independence versus disjointness:
  1. Independent events may occur together, and their intersection usually has positive probability.
  2. Disjoint events cannot occur together. If two disjoint events both have positive probability, they are not independent because (P(A\cap B)=0\neq P(A)P(B)).
• Example: In two fair coin tosses, (A={\text{first toss is heads}}) and (B={\text{second toss is heads}}). Since (P(A)=P(B)=1/2) and (P(A\cap B)=1/4), the events are independent.

V. Updated Probability — Information About an Event

A. Conditional Probability

Conditional probability measures the probability of one event after restricting attention to outcomes in another event.

• Definition:

P(A | B) = P(A ∩ B) / P(B), provided P(B) > 0

Here, (P(A\mid B)) is the probability of (A) given (B); (A\cap B) is their intersection; and (P(B)) is the probability of the conditioning event.

• Interpretation: Once (B) is known to have occurred, (B) becomes the reduced sample space, and (A\cap B) represents the favourable part.
• Rearranged form:

P(A ∩ B) = P(A | B)P(B)

• Asymmetry: Generally, (P(A\mid B)\neq P(B\mid A)); the two expressions use different conditioning information.
• Example: A fair die is rolled. Let (A={2,4,6}) and (B={4,5,6}). Then (A\cap B={4,6}), so

P(A | B) = (2/6)/(3/6) = 2/3

The condition (B) leaves three possible outcomes, two of which are even.

VI. Combinatorial Probability — Systematic Enumeration

A. Counting Sample Points

Counting techniques determine the number of outcomes in a sample space or event without listing every outcome individually.

• Addition principle: If one task can occur in (m) ways and a disjoint alternative in (n) ways, the total is (m+n).
• Multiplication principle: If a process has successive stages with (n_1,n_2,\ldots,n_k) choices, the total number of outcomes is

n₁n₂···nₖ

Here, (k) is the number of stages and (n_i) is the number of choices at stage (i).

• Permutations: The number of ordered arrangements of (r) objects selected from (n) distinct objects is

ⁿPᵣ = n!/(n − r)!

• Combinations: The number of unordered selections is

ⁿCᵣ = n!/[r!(n − r)!]

Here, (n) is the available number of objects, (r) is the number selected, and (n!) denotes (n(n-1)\cdots1).

• Example: A five-card hand chosen from 52 distinct cards ignores order, so the sample space contains (\binom{52}{5}=2,598,960) hands.
• Probability use: For equally likely outcomes, divide the number of favourable sample points by the total number of sample points.

VII. Degrees of Independence — Pairwise and Collective Conditions

A. Mutually and pairwise independent events

Pairwise independence checks events two at a time, whereas mutual independence requires every possible subcollection to factor into individual probabilities.

1. Pairwise independence: Events (A_1,\ldots,A_n) are pairwise independent if

P(Aᵢ ∩ Aⱼ) = P(Aᵢ)P(Aⱼ) for every i ≠ j

2. Mutual independence: For every selection of distinct indices (i_1,\ldots,i_k), where (2\leq k\leq n),

P(Aᵢ₁ ∩ ··· ∩ Aᵢₖ) = P(Aᵢ₁)···P(Aᵢₖ)

Here, (A_i) is the (i)-th event, (n) is the total number of events, and (k) is the size of a selected subcollection.

• Critical distinction: Mutual independence implies pairwise independence, but pairwise independence does not imply mutual independence.
• Example: Toss two fair coins. Let (A={\text{first is heads}}), (B={\text{second is heads}}), and (C={\text{both results match}}). Every pair is independent, but (P(A\cap B\cap C)=1/4), while (P(A)P(B)P(C)=1/8); therefore, the three events are not mutually independent.

VIII. Product Rules — Probabilities of Joint Occurrence

A. Multiplication theorem of probability for independent events

For independent events, the probability that all occur equals the product of their individual probabilities.

• Two-event theorem:

P(A ∩ B) = P(A)P(B)

• Multiple-event theorem: If (A_1,\ldots,A_n) are mutually independent, then

P(A₁ ∩ A₂ ∩ ··· ∩ Aₙ) = ∏ᵢ₌₁ⁿ P(Aᵢ)

Here, (A_i) is the (i)-th event, (n) is the number of events, and (\prod) denotes multiplication over all indices.

• Derivation from conditional probability:

P(A ∩ B) = P(A | B)P(B) = P(A)P(B)

The final equality follows because independence gives (P(A\mid B)=P(A)).

• At-least-one method: For independent events, it is often simpler to find the complement:

P(at least one occurs) = 1 − ∏ᵢ₌₁ⁿ[1 − P(Aᵢ)]

• Limitation: Multiplying marginal probabilities is invalid without independence; dependent events require conditional factors.

IX. Reverse Conditioning — Updating Beliefs from Evidence

A. Bayes’ theorem

Bayes’ theorem reverses conditional probability, using observed evidence to update the probability of a possible cause.

• Two-event form:

P(A | B) = P(B | A)P(A) / P(B), provided P(B) > 0

Here, (P(A)) is the prior probability, (P(B\mid A)) is the likelihood, (P(B)) is the evidence probability, and (P(A\mid B)) is the posterior probability.

• Partition form: If (A_1,\ldots,A_n) are disjoint, exhaustive events with positive probabilities, then

P(Aⱼ | B) =
[P(B | Aⱼ)P(Aⱼ)] /
[Σᵢ₌₁ⁿ P(B | Aᵢ)P(Aᵢ)]

Here, (A_j) is the hypothesis being updated, (B) is observed evidence, and the denominator follows from the law of total probability.

• Example: Suppose (1\%) of items are defective. A test is positive for (90\%) of defective items and (5\%) of nondefective items. Then

P(defective | positive)
= (0.90 × 0.01)/[(0.90 × 0.01) + (0.05 × 0.99)]
≈ 0.154

Thus, a positive result corresponds to about a (15.4\%) posterior probability of defect, demonstrating the strong influence of the low prior rate.