Unit 3 - Notes

ECE180

Unit 3: Operation on One Random Variable

1. Expected Value of a Random Variable

The expected value (or expectation) is the fundamental operator in signal processing and probability. It represents the "center of gravity" or the weighted average of a random variable.

1.1 Definition

If $X$ is a random variable, the expected value, denoted by $E[X]$ or $\bar{X}$ or $\mu$ , is defined as:

Discrete Case:

$E[X] = \sum_{i} x_i P_X(x_i)$
(Where $P_X(x_i)$ is the Probability Mass Function)
Continuous Case:

$E[X] = \int_{-\infty}^{\infty} x f_X(x) \, dx$
(Where $f_X(x)$ is the Probability Density Function)
Condition for Existence: The expectation exists only if the integral (or sum) converges absolutely: $\int_{-\infty}^{\infty} |x| f_X(x) dx < \infty$ .

1.2 Properties of Expectation

Constant: $E[c] = c$ , where $c$ is a constant.
Linearity: $E[aX + b] = aE[X] + b$ .
Additivity: $E[g(X) + h(X)] = E[g(X)] + E[h(X)]$ .

2. Function of a Random Variable

Often, we deal with a random variable $Y$ that is a function of another random variable $X$ , such that $Y = g(X)$ . To find the expected value of $Y$ , we do not necessarily need to find the PDF of $Y$ first. We can use the Law of the Unconscious Statistician (LOTUS).

2.1 The LOTUS Theorem

E[Y] = E[g(X)]

Discrete Case:

$E[g(X)] = \sum_{i} g(x_i) P_X(x_i)$
Continuous Case:

$E[g(X)] = \int_{-\infty}^{\infty} g(x) f_X(x) \, dx$

This theorem is powerful because it allows calculations entirely within the domain of $X$ .

3. Moments about the Origin

Moments describes the shape characteristics of a probability distribution function.

3.1 Definition

The $n$ -th moment of a random variable $X$ about the origin is denoted by $m_n$ and is defined as the expected value of the $n$ -th power of $X$ .

m_n = E[X^n] = \int_{-\infty}^{\infty} x^n f_X(x) \, dx

3.2 Notable Moments

Zeroth Moment ( $n=0$ ):
$m_0 = E[X^0] = E[1] = \int_{-\infty}^{\infty} f_X(x) dx = 1$
(Area under the PDF curve).
First Moment ( $n=1$ ):
$m_1 = E[X] = \mu$
This is the Mean or DC component of the signal.
Second Moment ( $n=2$ ):
$m_2 = E[X^2]$
This represents the Mean Square Value or the average power of the random variable.

4. Central Moments

Central moments are moments calculated about the mean ( $\mu$ ) rather than the origin (0). They describe the spread and shape of the distribution relative to its center.

4.1 Definition

The $n$ -th central moment, denoted by $\mu_n$ , is defined as:

\mu_n = E[(X - \mu)^n] = \int_{-\infty}^{\infty} (x - \mu)^n f_X(x) \, dx

4.2 Notable Central Moments

Zeroth Central Moment ( $n=0$ ):
$\mu_0 = E[(X-\mu)^0] = E[1] = 1$
First Central Moment ( $n=1$ ):
$\mu_1 = E[X - \mu] = E[X] - \mu = \mu - \mu = 0$
(The first central moment is always zero).
Second Central Moment ( $n=2$ ):
$\mu_2 = E[(X-\mu)^2] = \text{Variance} (\sigma^2)$

4.3 Relationship between Central and Raw Moments

Using binomial expansion on $E[(X-\mu)^n]$ , we can relate $\mu_n$ to $m_n$ .
For the second moment (Variance):

\sigma^2 = \mu_2 = E[X^2 - 2X\mu + \mu^2]

\sigma^2 = E[X^2] - 2\mu E[X] + \mu^2

\sigma^2 = m_2 - 2\mu(\mu) + \mu^2

\mathbf{\sigma^2 = m_2 - m_1^2 = E[X^2] - (E[X])^2}

5. Variance, Skewness, and Kurtosis

5.1 Variance ( $\sigma^2$ )

Variance measures the dispersion or "spread" of the random variable around the mean.

Definition: $\sigma_X^2 = \mu_2 = E[(X - \mu)^2]$
Standard Deviation ( $\sigma_X$ ): The square root of variance. It has the same units as $X$ .
Properties:
1. $\text{Var}(c) = 0$
2. $\text{Var}(aX + b) = a^2 \text{Var}(X)$

5.2 Skewness (Third Central Moment)

Skewness measures the asymmetry of the PDF around the mean.

Definition: $\mu_3 = E[(X-\mu)^3]$
Coefficient of Skewness ( $S_k$ ):
$S_k = \frac{\mu_3}{\sigma^3}$
Interpretation:
- $S_k = 0$ : Symmetric distribution (e.g., Gaussian).
- $S_k > 0$ : Positively skewed (Long tail to the right).
- $S_k < 0$ : Negatively skewed (Long tail to the left).

5.3 Kurtosis (Fourth Central Moment)

Kurtosis measures the "tailedness" or "peakedness" of the distribution.

Definition: $\mu_4 = E[(X-\mu)^4]$
Coefficient of Kurtosis ( $K_u$ ):
$K_u = \frac{\mu_4}{\sigma^4}$
Interpretation:
- Mesokurtic: $K_u = 3$ (Normal/Gaussian distribution).
- Leptokurtic: $K_u > 3$ (Sharper peak, heavy tails).
- Platykurtic: $K_u < 3$ (Flatter peak, light tails).

6. Chebyshev’s Inequality

Chebyshev’s inequality provides a bound on the probability that a random variable deviates from its mean by a certain amount. It applies to any distribution (discrete or continuous) as long as the variance is finite.

6.1 Statement

For a random variable $X$ with mean $\mu$ and variance $\sigma^2$ :

P(|X - \mu| \geq k\sigma) \leq \frac{1}{k^2}

where $k > 0$ is a real number.

6.2 Alternate Form

P(|X - \mu| < k\sigma) \geq 1 - \frac{1}{k^2}

Interpretation: The probability that $X$ lies within $k$ standard deviations of the mean is at least $1 - 1/k^2$ .

For $k=2$ : At least $75\%$ of data lies within $2\sigma$ of the mean.
For $k=3$ : At least $88.9\%$ of data lies within $3\sigma$ of the mean.

7. Characteristic Function

The characteristic function is the Fourier Transform of the PDF (with a sign reversal in the exponent). It always exists for any random variable.

7.1 Definition

Denoted by $\Phi_X(\omega)$ :

\Phi_X(\omega) = E[e^{j\omega X}] = \int_{-\infty}^{\infty} f_X(x) e^{j\omega x} \, dx

7.2 Properties

Maximum Value: $|\Phi_X(\omega)| \leq \Phi_X(0) = 1$ .
Origin: $\Phi_X(0) = 1$ .
Inversion Formula: We can recover the PDF from the characteristic function:
$f_X(x) = \frac{1}{2\pi} \int_{-\infty}^{\infty} \Phi_X(\omega) e^{-j\omega x} \, d\omega$

7.3 Moment Generation via Characteristic Function

We can find the $n$ -th moment ( $m_n$ ) by differentiating $\Phi_X(\omega)$ :

m_n = E[X^n] = (-j)^n \left[ \frac{d^n \Phi_X(\omega)}{d\omega^n} \right]_{\omega=0}

8. Moment Generating Function (MGF)

The MGF is closely related to the Laplace transform. Unlike the characteristic function, the MGF does not exist for all distributions (the integral may diverge).

8.1 Definition

Denoted by $M_X(t)$ :

M_X(t) = E[e^{tX}] = \int_{-\infty}^{\infty} f_X(x) e^{tx} \, dx

8.2 Moment Generation Property

The primary utility of the MGF is to calculate moments easily. The $n$ -th derivative of $M_X(t)$ evaluated at $t=0$ yields the $n$ -th moment about the origin.

m_n = E[X^n] = \left[ \frac{d^n M_X(t)}{dt^n} \right]_{t=0}

$m_1 = M_X'(0)$
$m_2 = M_X''(0)$

8.3 Relationship to Characteristic Function

\Phi_X(\omega) = M_X(j\omega)

9. Transformations of a Random Variable

If we know the PDF of a random variable $X$ ( $f_X(x)$ ) and we define a new variable $Y = g(X)$ , we aim to find the PDF of $Y$ ( $f_Y(y)$ ).

9.1 Method 1: Monotonic Transformations

If $g(X)$ is a one-to-one function (monotonic increasing or decreasing), there is a unique solution $x = g^{-1}(y)$ .

Formula:

f_Y(y) = f_X(x) \left| \frac{dx}{dy} \right|

where

x

must be replaced by

g^{-1}(y)

Steps:

Find the inverse relation $x = h(y)$ .
Calculate the derivative $\frac{dx}{dy}$ .
Substitute $x$ and the derivative into the formula.
Determine the valid range of $y$ .

9.2 Method 2: Non-Monotonic Transformations

If $g(X)$ is not one-to-one (e.g., $Y = X^2$ ), a single value of $y$ may correspond to multiple values of $x$ (roots). Let the roots be $x_1, x_2, ..., x_n$ .