Unit 4: Moments

ECAP790 1 min read

I. Orientation — Numerical Description of Random Variables

Moments are expected values of powers of random variables or their deviations from a reference point. They summarize a distribution’s location, dispersion, asymmetry, and tail behavior and also support probability bounds, transformations, and calculations involving several random variables.

Basic setting: Let (X) and (Y) be random variables on a common probability space, with probability mass functions in the discrete case or probability density functions in the continuous case.
Expectation convention: For a suitable function (g),
TEXT
```
  E[g(X)] = Σ g(x)pX(x)                 (discrete)
  E[g(X)] = ∫ g(x)fX(x) dx             (continuous)
```
Here (E) denotes expectation, (p_X(x)) is the probability mass function, and (f_X(x)) is the probability density function.
Existence condition: A moment is finite when the corresponding absolute expectation exists; for example, the (r)-th absolute moment requires (E[|X|^r]<\infty).
Principal measures:
- The mean (\mu=E[X]) measures location.
- The variance (\sigma^2=E[(X-\mu)^2]) measures dispersion.
- Higher moments describe asymmetry and tail concentration.
Joint setting: For several variables, moments include products such as (E[X^rY^s]), allowing dependence to be measured through covariance and related quantities.

II. Moment Measures — Description Around Zero and the Mean

A. Raw and central moments

Raw moments measure powers about zero, whereas central moments measure powers about the distribution’s mean.

Raw moment: The (r)-th raw moment, also called the moment about the origin, is
TEXT
```
  μ′r = E[X^r],    r = 1, 2, 3, ...
```
Here (\mu'_r) denotes the (r)-th raw moment and (r) is a positive integer. In particular, (\mu'_1=E[X]=\mu).
Central moment: The (r)-th central moment is
TEXT
```
  μr = E[(X - μ)^r]
```
Here (\mu_r) is the (r)-th central moment and (\mu=E[X]). The first central moment is (0), and the second is the variance:
TEXT
```
  μ1 = 0,    μ2 = σ²
```
Conversion from raw moments: Expanding by the binomial theorem gives
TEXT
```
  μr = Σ(k=0 to r) C(r,k)(-μ)^(r-k) μ′k
```
Here (C(r,k)) is the binomial coefficient and (\mu'_0=E[1]=1). Important cases are
TEXT
```
  μ2 = μ′2 - μ²
  μ3 = μ′3 - 3μμ′2 + 2μ³
  μ4 = μ′4 - 4μμ′3 + 6μ²μ′2 - 3μ⁴
```
Standardized moments: Removing units permits comparison between distributions:
TEXT
```
  skewness = μ3 / σ³
  kurtosis = μ4 / σ⁴
```
Skewness indicates asymmetry. Kurtosis measures standardized fourth-moment concentration; excess kurtosis is (\mu_4/\sigma^4-3).
Effect of transformation: If (Z=aX+b), then (E[Z]=a\mu+b), while for (r\ge 1),
TEXT
```
  E[(Z - E[Z])^r] = a^r μr
```
Thus translation does not change central moments of order at least two, while scaling multiplies them by (a^r).

III. Probability Bounds — Control Through Mean and Variance

A. Chebyshev Inequality

Chebyshev Inequality bounds the probability of a random variable lying far from its mean using only a finite variance.

Formal statement: If (E[X]=\mu) and (\operatorname{Var}(X)=\sigma^2<\infty), then for every (\varepsilon>0),
TEXT
```
  P(|X - μ| ≥ ε) ≤ σ² / ε²
```
Equivalently, for (k>0),
TEXT
```
  P(|X - μ| ≥ kσ) ≤ 1/k²
  P(|X - μ| < kσ) ≥ 1 - 1/k²
```
Here (P) denotes probability, (\varepsilon) is a positive distance, and (k) is the number of standard deviations.
Derivation: Apply Markov’s inequality to the nonnegative variable ((X-\mu)^2):
TEXT
```
  P((X - μ)² ≥ ε²)
      ≤ E[(X - μ)²] / ε²
      = σ² / ε²
```
Interpretation: At least (75\%) of probability lies within (2) standard deviations because (1-1/2^2=3/4); at least (88.89\%) lies within (3) standard deviations.
Scope and limitation: The inequality applies to discrete, continuous, symmetric, or skewed distributions, but it is often conservative because it uses only (\mu) and (\sigma^2).
Application to sample means: For independent identically distributed variables with variance (\sigma^2), the sample mean (\bar X_n) has variance (\sigma^2/n), so
TEXT
```
  P(|X̄n - μ| ≥ ε) ≤ σ² / (nε²)
```
This bound tends to zero as (n\to\infty), establishing the weak law of large numbers under finite variance.

IV. Joint Distributions — Measuring Interaction

A. Moments of Two or More Random Variables

Joint moments describe the combined behavior and dependence of random variables defined on the same probability space.

Joint raw moments: For nonnegative integers (r) and (s),
TEXT
```
  μ′r,s = E[X^r Y^s]
```
For continuous variables this equals
TEXT
```
  E[X^r Y^s] = ∬ x^r y^s fX,Y(x,y) dx dy
```
Here (f_{X,Y}) is the joint density. The discrete form replaces the integral with a double sum using the joint mass function.

Joint central moments: With (\mu_X=E[X]) and (\mu_Y=E[Y]),

TEXT

  μr,s = E[(X - μX)^r (Y - μY)^s]

The case (r=s=1) is covariance:

TEXT

  Cov(X,Y) = E[(X - μX)(Y - μY)]
           = E[XY] - E[X]E[Y]

Correlation: Standardized covariance is
TEXT
```
  ρX,Y = Cov(X,Y) / (σX σY)
```
Here (\rho_{X,Y}) is the correlation coefficient and (\sigma_X,\sigmaY>0) are standard deviations. It satisfies (-1\le\rho{X,Y}\le1).
Independence: If (X) and (Y) are independent and the moments exist, then
TEXT
```
  E[X^r Y^s] = E[X^r]E[Y^s]
```
Independence therefore implies zero covariance, but zero covariance does not generally imply independence.
Several variables: For a random vector (\mathbf X=(X_1,\ldots,Xn)^T), the covariance matrix has entries
TEXT
```
  Σij = Cov(Xi, Xj)
```
Here (\Sigma{ij}) is the entry in row (i), column (j). The matrix is symmetric and positive semidefinite.

V. Aggregation — Moments Under Addition

A. Moments of Sums of Random Variables

Moments of a sum are obtained by linearity of expectation and by expanding powers of the sum, with dependence represented by mixed moments.

First moment: For (S=\sum_{i=1}^n X_i),
TEXT
```
  E[S] = Σ(i=1 to n) E[Xi]
```
This identity requires no independence.
Second central moment: The variance of the sum is
TEXT
```
  Var(S) = Σ(i=1 to n) Var(Xi)
           + 2Σ(i<j) Cov(Xi,Xj)
```
If the variables are pairwise uncorrelated, all covariance terms vanish. In particular, independent variables satisfy
TEXT
```
  Var(S) = Σ(i=1 to n) Var(Xi)
```
Higher raw moments: For two variables, the binomial expansion gives
TEXT
```
  E[(X + Y)^r]
    = Σ(k=0 to r) C(r,k) E[X^k Y^(r-k)]
```
Independence allows each mixed moment to factor into (E[X^k]E[Y^{r-k}]).
Identically distributed sums: If (X_1,\ldots,X_n) are independent with common mean (\mu) and variance (\sigma^2), then
TEXT
```
  E[S] = nμ,    Var(S) = nσ²
  E[X̄n] = μ,   Var(X̄n) = σ²/n
```
Thus totals become more variable in absolute terms, while averages become more concentrated.

VI. Transform Representation — Encoding All Moments

A. Moment Generating Function

The Moment Generating Function packages a distribution’s raw moments into a single function.

Definition: The MGF of (X) is
TEXT
```
  MX(t) = E[e^(tX)]
```
Here (M_X(t)) denotes the MGF and (t) is a real parameter. It is useful when finite on an open interval containing (t=0).

Discrete and continuous forms:

TEXT

  MX(t) = Σ e^(tx)pX(x)
  MX(t) = ∫ e^(tx)fX(x) dx

Moment expansion: Expanding the exponential formally gives
TEXT
```
  MX(t) = 1 + μ′1 t + μ′2 t²/2! + μ′3 t³/3! + ...
```
Consequently, moments are encoded as coefficients of powers of (t).
Existence issue: An MGF may fail to exist for nonzero (t), even when every positive integer moment exists. Therefore, moment calculations through MGFs require finiteness near zero.
Example: If (X\sim N(\mu,\sigma^2)), then
TEXT
```
  MX(t) = exp(μt + σ²t²/2)
```
Differentiation at zero gives (E[X]=\mu) and (E[X^2]=\mu^2+\sigma^2).

VII. Operational Rules — Using the MGF

A. Properties of moment generating function

The principal properties of the MGF allow moments, transformations, sums, and distributions to be handled algebraically.

Value at the origin:
TEXT
```
  MX(0) = E[1] = 1
```
Derivatives and moments: When differentiation under expectation is justified,
TEXT
```
  MX^(r)(t) = E[X^r e^(tX)]
  MX^(r)(0) = E[X^r] = μ′r
```
Here (M_X^{(r)}) is the (r)-th derivative.
Linear transformation: If (Y=aX+b), then
TEXT
```
  MY(t) = e^(bt) MX(at)
```
This follows directly from (E[e^{t(aX+b)}]).
Independent sums: If (X_1,\ldots,X_n) are independent and (S=\sum X_i), then
TEXT
```
  MS(t) = Π(i=1 to n) MXi(t)
```
For identically distributed variables, (M_S(t)=[M_X(t)]^n). Without independence, this factorization generally fails.
Uniqueness: If two MGFs are finite and equal throughout an open interval containing zero, their corresponding distributions are identical.
Domain and convexity: The set of (t)-values for which (M_X(t)<\infty) is an interval containing zero, possibly only at its boundary; wherever finite, (M_X(t)) is convex.

VIII. Logarithmic Moment Measures — Additive Distribution Characteristics

A. Cumulants

Cumulants are coefficients obtained from the logarithm of the MGF and are especially useful because they add for independent random variables.

Cumulant generating function:
TEXT
```
  KX(t) = log MX(t)
```
Here (K_X(t)) is the cumulant generating function. The (r)-th cumulant is
TEXT
```
  κr = KX^(r)(0)
```
where (\kappa_r) denotes the (r)-th cumulant.
First four cumulants:
TEXT
```
  κ1 = μ
  κ2 = μ2 = σ²
  κ3 = μ3
  κ4 = μ4 - 3μ2²
```
Thus the first cumulant is the mean, the second is the variance, the third is the third central moment, and the fourth measures departure from normal fourth-moment behavior.
Independent additivity: If (S=X+Y) for independent (X) and (Y), then
TEXT
```
  KS(t) = KX(t) + KY(t)
  κr(S) = κr(X) + κr(Y)
```
This converts multiplication of MGFs into addition.

Transformation rule: For (Y=aX+b),

TEXT

  κ1(Y) = aκ1(X) + b
  κr(Y) = a^r κr(X),    r ≥ 2

Normal distribution: For (X\sim N(\mu,\sigma^2)),
TEXT
```
  KX(t) = μt + σ²t²/2
```
Hence (\kappa_1=\mu), (\kappa_2=\sigma^2), and every cumulant of order (r\ge3) is zero.

Unit 4: Moments

Table of Contents

I. Orientation — Numerical Description of Random Variables

II. Moment Measures — Description Around Zero and the Mean

A. Raw and central moments

III. Probability Bounds — Control Through Mean and Variance

A. Chebyshev Inequality

IV. Joint Distributions — Measuring Interaction

A. Moments of Two or More Random Variables

V. Aggregation — Moments Under Addition

A. Moments of Sums of Random Variables

VI. Transform Representation — Encoding All Moments

A. Moment Generating Function

VII. Operational Rules — Using the MGF

A. Properties of moment generating function

VIII. Logarithmic Moment Measures — Additive Distribution Characteristics

A. Cumulants

Did this save you a night before the exam?