Back to Subject

Practice MCQ

Unit 1 - Notes

ECE180

Unit 1: Probability and Random Variable

1. Fundamentals of Set Theory

Probability theory relies heavily on the mathematical framework of set theory.

Basic Definitions

Set: A collection of distinct objects.
Element: An object belonging to a set. Notation: $w \in A$ (element $w$ belongs to set $A$ ).
Subset: If every element of $A$ is also in $B$ , then $A \subset B$ .
Universal Set ( $\Omega$ or $S$ ): The set containing all objects under consideration. In probability, this corresponds to the Sample Space.
Empty (Null) Set ( $\phi$ ): A set containing no elements.

Set Operations

Union ( $A \cup B$ ): The set of elements in $A$ , or in $B$ , or in both. (Logical OR).
Intersection ( $A \cap B$ ): The set of elements common to both $A$ and $B$ . (Logical AND).
Complement ( $\bar{A}$ or $A^c$ ): The set of elements in $\Omega$ that are not in $A$ .
Difference ( $A - B$ ): Elements in $A$ but not in $B$ . Equivalent to $A \cap \bar{B}$ .

Algebraic Laws of Sets

Commutative: $A \cup B = B \cup A$ ; $A \cap B = B \cap A$
Associative: $(A \cup B) \cup C = A \cup (B \cup C)$
Distributive: $A \cap (B \cup C) = (A \cap B) \cup (A \cap C)$
De Morgan’s Laws:
- $\overline{A \cup B} = \bar{A} \cap \bar{B}$
- $\overline{A \cap B} = \bar{A} \cup \bar{B}$

2. Experiments and Sample Spaces

Experiments

An experiment is a process that produces an outcome.

Deterministic Experiment: An experiment where the outcome can be predicted with certainty (e.g., Ohm’s law verification).
Random (Stochastic) Experiment: An experiment where the outcome cannot be predicted with certainty, even if the experiment is repeated under identical conditions (e.g., tossing a coin, measuring noise voltage).

Sample Space ( $\Omega$ )

The set of all possible outcomes of a random experiment.

Each outcome is a sample point.
Example: Rolling a die, $\Omega = \{1, 2, 3, 4, 5, 6\}$ .

Types of Sample Spaces

Discrete Sample Space: Contains a finite or countably infinite number of sample points.
- Example: Tossing a coin ( $H, T$ ) or counting the number of calls arriving at a switch (0, 1, 2...).
Continuous Sample Space: Contains an uncountably infinite number of sample points (a continuum).
- Example: Measuring the exact time of arrival of a signal ( $0 < t < \infty$ ) or measuring temperature.

3. Events

An Event is a subset of the sample space $\Omega$ .

Simple Event: An event containing only one sample point.
Compound Event: An event containing more than one sample point.
Sure Event: The sample space $\Omega$ itself (probability = 1).
Impossible (Null) Event: The empty set $\phi$ (probability = 0).

Relationships Between Events

Mutually Exclusive (Disjoint) Events: Two events and are mutually exclusive if they cannot occur simultaneously.
- $A \cap B = \phi$
Exhaustive Events: A collection of events is exhaustive if their union equals the sample space.
- $\bigcup_{i=1}^n A_i = \Omega$

4. Probability Definitions and Axioms

1. Classical (A Priori) Definition

Based on the assumption of equally likely outcomes. If an experiment has $N$ total outcomes and $N_A$ outcomes favorable to event $A$ :

P(A) = \frac{N_A}{N}

Limitation: Only valid for finite sample spaces with equally likely outcomes.

2. Relative Frequency (Empirical) Definition

Based on repeating an experiment $n$ times. If event $A$ occurs $n_A$ times:

P(A) = \lim_{n \to \infty} \frac{n_A}{n}

Limitation: Cannot perform an experiment infinite times; statistical regularity is assumed.

3. Axiomatic Definition (Kolmogorov's Axioms)

Let $\Omega$ be a sample space. Probability $P$ is a function that assigns a real number to every event $A$ , satisfying:

Axiom 1 (Non-negativity): $P(A) \geq 0$ for any event $A$ .
Axiom 2 (Normalization): $P(\Omega) = 1$ .
Axiom 3 (Additivity): If $A$ and $B$ are mutually exclusive ( $A \cap B = \phi$ ), then:
$P(A \cup B) = P(A) + P(B)$
(This extends to infinite sequences of mutually exclusive events).

5. Mathematical Model of Experiments

A rigorous mathematical model for a random experiment consists of a triplet $(\Omega, \mathcal{F}, P)$ , often called a Probability Space.

Sample Space ( $\Omega$ ): The set of all outcomes.
Sigma-Algebra / Field ( $\mathcal{F}$ ): A collection of subsets (events) of $\Omega$ that is closed under complementation and countable unions. This defines the "measurable" events.
Probability Measure ( $P$ ): A function $P: \mathcal{F} \to [0, 1]$ satisfying the three axioms listed above.

6. Joint, Conditional, and Total Probability

Joint Probability

The probability that two events $A$ and $B$ occur simultaneously.

P(A \cap B) \text{ or } P(AB)

Conditional Probability

The probability of event $A$ occurring given that event $B$ has already occurred (where $P(B) > 0$ ).

P(A|B) = \frac{P(A \cap B)}{P(B)}

Property:

P(A \cap B) = P(A|B)P(B) = P(B|A)P(A)

Total Probability Theorem

If $B_1, B_2, ..., B_n$ is a set of mutually exclusive and exhaustive events (a partition of $\Omega$ ), then for any event $A$ :

P(A) = \sum_{i=1}^n P(A \cap B_i)

Using conditional probability:

P(A) = \sum_{i=1}^n P(A|B_i)P(B_i)

Concept: This breaks a complex probability

P(A)

into smaller, manageable conditional parts based on the partition

B

7. Bayes' Theorem and Independence

Bayes' Theorem

Used to find the "reverse" probability. If we know $P(A|B)$ , Bayes' theorem allows us to find $P(B|A)$ . Using the partition $B_1...B_n$ from the Total Probability Theorem:

P(B_k|A) = \frac{P(A|B_k)P(B_k)}{P(A)}

Expanded form:

P(B_k|A) = \frac{P(A|B_k)P(B_k)}{\sum_{i=1}^n P(A|B_i)P(B_i)}

$P(B_k)$ is the Prior probability.
$P(B_k|A)$ is the Posterior probability.

Independent Events

Two events $A$ and $B$ are statistically independent if the occurrence of one does not affect the probability of the other.

Condition: $P(A \cap B) = P(A)P(B)$
Equivalent: $P(A|B) = P(A)$ and $P(B|A) = P(B)$

Note: Do not confuse "Mutually Exclusive" with "Independent". Mutually exclusive events are highly dependent (if one happens, the other cannot).

8. Bernoulli's Trials

A Bernoulli trial is a random experiment with exactly two possible outcomes: "Success" and "Failure".

Properties:

There are only two outcomes.
The probability of success is $p$ .
The probability of failure is $q = 1 - p$ .
Trials are independent.

Binomial Probability Law:
The probability of obtaining exactly $k$ successes in $n$ independent Bernoulli trials is:

P(k) = \binom{n}{k} p^k q^{n-k}

Where

\binom{n}{k} = \frac{n!}{k!(n-k)!}

9. The Random Variable (RV)

Definition

A Random Variable $X$ is a real-valued function that maps every outcome $\omega$ in the sample space $\Omega$ to a real number.

X: \Omega \to \mathbb{R}

Example: Tossing a coin twice.

\Omega = \{HH, HT, TH, TT\}

. Let

X

be the number of heads.

X(HH) = 2, X(HT) = 1, X(TH) = 1, X(TT) = 0

Conditions for a Function to be a Random Variable

For a function $X(\omega)$ to be a valid random variable:

It must be single-valued (each outcome maps to exactly one number).
For every real number $x$ , the set $\{ \omega : X(\omega) \leq x \}$ must be an event (i.e., it must belong to the field $\mathcal{F}$ so that we can calculate its probability).
$P(X = -\infty) = 0$ and $P(X = +\infty) = 0$ .

10. Types of Random Variables

1. Discrete Random Variable

Definition: An RV that takes on a countable number of distinct values (finite or countably infinite).
Range: $R_X = \{x_1, x_2, x_3, ...\}$
Description: Described by a Probability Mass Function (PMF), denoted .
- $p_X(x_i) = P(X = x_i)$
Example: Outcome of a die roll, number of defective items in a batch.

2. Continuous Random Variable

Definition: An RV that takes on an infinite number of values within a continuous interval (uncountable).
Range: An interval on the real line, e.g., $[a, b]$ or $(-\infty, \infty)$ .
Description: Described by a Probability Density Function (PDF), denoted .
- Probability at a specific point is 0: $P(X=c) = 0$ .
- Probability is calculated over intervals: $P(a < X < b) = \int_{a}^{b} f_X(x) dx$ .
Example: Noise voltage, temperature, height of students.

3. Mixed Random Variable

Definition: An RV for which the Cumulative Distribution Function (CDF) exhibits both jump discontinuities (discrete behavior) and continuous growth intervals.
It contains outcomes that occur with non-zero probability (discrete points) and ranges where probability is spread continuously.
Representation: The density function involves Dirac delta functions ( $\delta(x)$ ) at the discrete points.
$f_X(x) = f_{discrete}(x) + f_{continuous}(x)$
Example: The waiting time at a traffic light (probability of waiting 0 seconds is non-zero if the light is green, but if red, the wait time is continuous).