Unit 4 - Notes

ECE180

Unit 4: Multiple Random Variables

1. Vector Random Variables

A Vector Random Variable (or a multidimensional random variable) is a vector function $\mathbf{X}$ that maps outcomes $\zeta$ from a sample space $S$ into an $N$ -dimensional Euclidean space $\mathbb{R}^N$ .

Notation: $\mathbf{X} = (X_1, X_2, \dots, X_N)$ where each $X_i$ is a random variable.
Significance: In many engineering problems, a single number is insufficient to describe the outcome of an experiment (e.g., measuring both the magnitude and phase of a signal, or the coordinates of a particle).
Events: An event is defined by a region $D$ in the $N$ -dimensional space. The probability of the event is $P(\mathbf{X} \in D)$ .

2. Joint Distribution Function (CDF)

For two random variables $X$ and $Y$ , the Joint Cumulative Distribution Function (CDF) is defined as the probability that $X \le x$ and $Y \le y$ simultaneously.

F_{XY}(x, y) = P(X \le x \cap Y \le y)

Properties of Joint CDF

Non-decreasing: $F_{XY}(x, y)$ is a non-decreasing function of both $x$ and $y$ .
Bounds:
- $0 \le F_{XY}(x, y) \le 1$
- $F_{XY}(-\infty, -\infty) = F_{XY}(-\infty, y) = F_{XY}(x, -\infty) = 0$
- $F_{XY}(\infty, \infty) = 1$
Right-Continuous: The function is continuous from the right.
Probability of a Rectangle: To find $P(x_1 < X \le x_2, y_1 < Y \le y_2)$ :
$P(x_1 < X \le x_2, y_1 < Y \le y_2) = F_{XY}(x_2, y_2) - F_{XY}(x_1, y_2) - F_{XY}(x_2, y_1) + F_{XY}(x_1, y_1)$

3. Joint Density Function (PDF)

For continuous random variables, the Joint Probability Density Function (PDF) is the partial derivative of the joint CDF.

f_{XY}(x, y) = \frac{\partial^2 F_{XY}(x, y)}{\partial x \partial y}

Properties of Joint PDF

Non-negative: $f_{XY}(x, y) \ge 0$ for all $x, y$ .
Normalization: The volume under the density surface is unity.
$\int_{-\infty}^{\infty} \int_{-\infty}^{\infty} f_{XY}(x, y) \, dx \, dy = 1$
CDF from PDF:
$F_{XY}(x, y) = \int_{-\infty}^{y} \int_{-\infty}^{x} f_{XY}(u, v) \, du \, dv$
Probability of a Region: The probability that the point $(X, Y)$ falls within a specific region $D$ in the $xy$ -plane is:
$P((X, Y) \in D) = \iint_D f_{XY}(x, y) \, dx \, dy$

4. Marginal Distribution and Density Functions

Marginal functions allow us to obtain the properties of a single random variable from the joint probability model by "integrating out" or "summing out" the other variables.

Marginal CDF

Marginal CDF of X: $F_X(x) = F_{XY}(x, \infty)$
Marginal CDF of Y: $F_Y(y) = F_{XY}(\infty, y)$

Marginal PDF

To find the density of $X$ alone, we integrate the joint density over the entire range of $Y$ .

Marginal PDF of X:
$f_X(x) = \int_{-\infty}^{\infty} f_{XY}(x, y) \, dy$
Marginal PDF of Y:
$f_Y(y) = \int_{-\infty}^{\infty} f_{XY}(x, y) \, dx$

5. Conditional Distribution and Density Functions

These functions describe the behavior of one random variable assuming the other has taken on a specific value.

Conditional PDF

The conditional PDF of $X$ given $Y=y$ is defined as:

f_{X|Y}(x|y) = \frac{f_{XY}(x, y)}{f_Y(y)}, \quad \text{provided } f_Y(y) > 0

Similarly, for $Y$ given $X=x$ :

f_{Y|X}(y|x) = \frac{f_{XY}(x, y)}{f_X(x)}, \quad \text{provided } f_X(x) > 0

Properties

Normalization: $\int_{-\infty}^{\infty} f_{X|Y}(x|y) \, dx = 1$ . It behaves exactly like a standard single-variable PDF.
Chain Rule of Probability: $f_{XY}(x, y) = f_{X|Y}(x|y) f_Y(y) = f_{Y|X}(y|x) f_X(x)$ .

Conditional CDF

F_{X|Y}(x|y) = \frac{\int_{-\infty}^{x} f_{XY}(u, y) \, du}{f_Y(y)}

6. Statistical Independence

Two random variables $X$ and $Y$ are statistically independent if the occurrence of an event associated with one variable has no influence on the probability of an event associated with the other.

Conditions for Independence

If $X$ and $Y$ are independent, the following equivalent conditions hold:

PDF Factorization:
$f_{XY}(x, y) = f_X(x) \cdot f_Y(y)$
CDF Factorization:
$F_{XY}(x, y) = F_X(x) \cdot F_Y(y)$
Conditional Equals Marginal:
$f_{X|Y}(x|y) = f_X(x) \quad \text{and} \quad f_{Y|X}(y|x) = f_Y(y)$

Note: If the joint support (the region where $f_{XY} > 0$ ) is not a rectangle (e.g., a triangle or circle), the variables are dependent, even if the function looks separable.

7. Sum of Two Random Variables

Let $W = X + Y$ . We seek the statistics of $W$ .

Using the PDF (Convolution)

If $X$ and $Y$ are independent continuous random variables, the PDF of their sum $W$ is the convolution of their individual PDFs.

f_W(w) = f_X(x) * f_Y(y) = \int_{-\infty}^{\infty} f_X(x) f_Y(w - x) \, dx

Alternatively:

f_W(w) = \int_{-\infty}^{\infty} f_Y(y) f_X(w - y) \, dy

Sum of Several Random Variables

If $Y = X_1 + X_2 + \dots + X_N$ where all $X_i$ are independent:

The PDF of $Y$ is the convolution of the PDFs of all $X_i$ .
$f_Y(y) = f_{X_1} * f_{X_2} * \dots * f_{X_N}$
Mean: $\mu_Y = \sum \mu_{X_i}$
Variance: $\sigma_Y^2 = \sum \sigma_{X_i}^2$

8. Central Limit Theorem (CLT)

The CLT states that the probability distribution of the sum (or average) of a large number of independent, identically distributed (i.i.d.) random variables approaches a Gaussian (Normal) distribution, regardless of the original distribution shape.

Statement

Let $X_1, X_2, \dots, X_N$ be $N$ independent random variables with mean $\mu$ and variance $\sigma^2$ . Let $Y_N = \sum_{i=1}^N X_i$ .

As $N \to \infty$ :

$Y_N$ approaches a Normal distribution.
Mean of Sum: $\bar{Y}_N = N\mu$
Variance of Sum: $\sigma_{Y_N}^2 = N\sigma^2$

The Normalized Form

The random variable $Z_N$ :

Z_N = \frac{Y_N - N\mu}{\sigma\sqrt{N}}

approaches the Standard Normal Distribution

\mathcal{N}(0, 1)

N \to \infty

9. Expected Value of a Function of Random Variables

If $g(X, Y)$ is a function of two random variables $X$ and $Y$ , the expected value is:

E[g(X, Y)] = \int_{-\infty}^{\infty} \int_{-\infty}^{\infty} g(x, y) f_{XY}(x, y) \, dx \, dy

Linearity of Expectation

The expectation operator is linear:

E[aX + bY + c] = aE[X] + bE[Y] + c

This holds regardless of whether

X

and

Y

are independent.

10. Joint Moments

Joint moments describe the statistical relationship between two variables.

Joint Moments about the Origin ( $m_{nk}$ )

The $(n, k)$ -th joint moment is defined as:

m_{nk} = E[X^n Y^k] = \int_{-\infty}^{\infty} \int_{-\infty}^{\infty} x^n y^k f_{XY}(x, y) \, dx \, dy

$m_{10} = E[X]$ (Mean of X, $\mu_X$ )
$m_{01} = E[Y]$ (Mean of Y, $\mu_Y$ )
$m_{11} = E[XY]$ : This is the Correlation of and .
- If $E[XY] = 0$ , $X$ and $Y$ are orthogonal.

Joint Central Moments ( $\mu_{nk}$ )

These are moments taken about the respective means ( $\bar{X}$ and $\bar{Y}$ ).

\mu_{nk} = E[(X - \bar{X})^n (Y - \bar{Y})^k]

$\mu_{20} = E[(X-\bar{X})^2] = \sigma_X^2$ (Variance of X)
$\mu_{02} = E[(Y-\bar{Y})^2] = \sigma_Y^2$ (Variance of Y)
$\mu_{11} = E[(X-\bar{X})(Y-\bar{Y})]$ : This is the Covariance, denoted $\text{Cov}(X, Y)$ or $\sigma_{XY}$ .

Covariance Formulas

\text{Cov}(X, Y) = E[XY] - E[X]E[Y] = m_{11} - m_{10}m_{01}

If $\text{Cov}(X, Y) = 0$ , the variables are Uncorrelated.
Relationship to Independence:
- Independence $\implies$ Uncorrelated.
- Uncorrelated $\does not \implies$ Independence (except for Joint Gaussian RVs).

Correlation Coefficient ( $\rho$ )

A normalized measure of linear dependence between $X$ and $Y$ .

\rho_{XY} = \frac{\text{Cov}(X, Y)}{\sigma_X \sigma_Y}

$-1 \le \rho_{XY} \le 1$
$\rho = 1$ : Perfect positive linear relationship.
$\rho = -1$ : Perfect negative linear relationship.
$\rho = 0$ : Uncorrelated.