Unit 6 - Notes

ECE180

Unit 6: Noise Sources & Information Theory

1. Noise Sources and Characterization

Noise is defined as any unwanted electrical signal that interferes with the transmission, processing, or reception of information signals.

1.1 Resistive / Thermal Noise (Johnson-Nyquist Noise)

Thermal noise is generated by the random thermal motion of electrons inside a resistor. It exists in all resistive components regardless of applied voltage.

Origin: Random collision of electrons with atoms due to thermal energy.
Characteristics: It is White Noise (constant power spectral density) up to extremely high frequencies (near optical).
Mean Value: The average voltage is zero ( $\overline{v(t)} = 0$ ).
Variance (Mean Square Value):
For a resistor $R$ at temperature $T$ (Kelvin) over a bandwidth $B$ (Hz):

Where:
- $k$ : Boltzmann’s constant ( $1.38 \times 10^{-23} \, \text{J/K}$ )
- $T$ : Absolute temperature in Kelvin
- $R$ : Resistance in Ohms
- $B$ : Bandwidth in Hz
Circuit Models:
1. Thevenin Equivalent: A noise voltage source $V_{rms}$ in series with a noiseless resistor $R$ .
2. Norton Equivalent: A noise current source $I_{rms}$ in parallel with a noiseless resistor $R$ .
  $I_{rms}^2 = \overline{i_n^2} = \frac{4kTB}{R}$
Maximum Power Transfer:
The maximum noise power ( $P_n$ ) delivered to a matched load ( $R_L = R$ ) is:

$P_n = kTB$
Power Spectral Density (PSD):
- Two-sided PSD: $S_v(f) = 2kTR$ (V²/Hz)
- One-sided PSD: $S_v(f) = 4kTR$ (V²/Hz) for $f \ge 0$

1.2 Arbitrary Noise Sources

Apart from thermal noise, other common noise sources include:

Shot Noise:
- Arises in active devices (diodes, transistors) due to the discrete nature of charge carriers (electrons/holes) crossing a potential barrier.
- Formula: $I_{sh}^2 = 2qI_{dc}B$
- Where $q$ is electron charge and $I_{dc}$ is the DC current.
Flicker Noise (1/f Noise):
- Dominant at low frequencies.
- PSD is inversely proportional to frequency ( $S(f) \propto 1/f$ ).
Burst Noise (Popcorn Noise):
- Sudden step-like transitions in current/voltage levels in semiconductors.

1.3 Effective Noise Temperature ( $T_e$ )

A convenient way to characterize the noise performance of a device or system.

Definition: The effective noise temperature of a device is the temperature $T_e$ at which a hypothetical resistor (equal to the input resistance of the device) would generate the same thermal noise power as the device adds internally.
Formula:
$T_e = \frac{P_{noise\_output}}{kGB} - T_{source}$
Usually expressed as:
$P_{internal} = k T_e B G$
Where $G$ is the power gain.

1.4 Noise Equivalent Bandwidth ( $B_{eq}$ )

Actual filters do not have perfectly rectangular frequency responses. $B_{eq}$ equates an actual filter to an ideal rectangular filter.

Definition: The bandwidth of an ideal rectangular filter that passes the same noise power as the actual filter, assuming the maximum gain of both filters is the same.
Formula:
- $H(f)$ : Transfer function of the system.
- $H(f)_{max}$ : Maximum magnitude of the transfer function.

2. Noise Figure Characterization

2.1 Average Noise Figure ( $F$ )

A figure of merit measuring the degradation of the Signal-to-Noise Ratio (SNR) as a signal passes through a system.

Definition: The ratio of the input SNR to the output SNR.
$F = \frac{SNR_{in}}{SNR_{out}} = \frac{S_i / N_i}{S_o / N_o}$
Since $S_o = G S_i$ and $N_o = G(N_i + N_{added})$ , where $G$ is gain:
$F = \frac{N_o}{G N_i}$
Ideally: $F = 1$ (0 dB) for a noiseless system.
Relation to Noise Temperature:
$F = 1 + \frac{T_e}{T_0}$
Where $T_0$ is the standard reference temperature (usually 290 K).

2.2 Average Noise Figure of Cascaded Networks (Friis Formula)

When amplifiers are connected in series (cascade), the total noise figure depends heavily on the first stage.

Scenario: Stage 1 (Gain $G_1$ , Noise Figure $F_1$ ) followed by Stage 2 (Gain $G_2$ , Noise Figure $F_2$ ), etc.
Formula:
$F_{total} = F_1 + \frac{F_2 - 1}{G_1} + \frac{F_3 - 1}{G_1 G_2} + \dots + \frac{F_n - 1}{G_1 G_2 \dots G_{n-1}}$
Key Insight: The noise contribution of downstream stages is divided by the total gain of previous stages. Therefore, the first stage needs low noise ( $F_1$ ) and high gain ( $G_1$ ) to minimize the overall system noise figure.

3. Narrow Band Noise

3.1 Definition

Narrow band noise is noise whose spectral content is concentrated within a narrow bandwidth $B$ around a center frequency $f_c$ , such that $B \ll f_c$ . This is typical in bandpass filter outputs in communication receivers.

3.2 Quadrature Representation

Narrow band noise $n(t)$ can be represented in terms of its in-phase and quadrature components:

n(t) = n_c(t)\cos(2\pi f_c t) - n_s(t)\sin(2\pi f_c t)

$n_c(t)$ : In-phase component (varies slowly).
$n_s(t)$ : Quadrature component (varies slowly).

3.3 Properties of Narrow Band Noise

If $n(t)$ is a zero-mean, stationary, Gaussian random process with PSD $S_n(f)$ :

Mean: $E[n_c(t)] = E[n_s(t)] = 0$ .
Variance: The power in the components equals the total noise power.
$E[n^2(t)] = E[n_c^2(t)] = E[n_s^2(t)] = \sigma^2$
PSD Relationship: The PSD of $n_c(t)$ and $n_s(t)$ is related to the low-pass equivalent of the bandpass PSD $S_n(f)$ .
Statistical Independence: At the same time instant $t$ , $n_c(t)$ and $n_s(t)$ are uncorrelated (and independent if Gaussian).
Envelope and Phase:
Representing in polar form: .
- Envelope $R(t)$ : Follows a Rayleigh Distribution.
- Phase $\phi(t)$ : Follows a Uniform Distribution over $[0, 2\pi]$ .

4. Information Theory Fundamentals

4.1 Information Content (Self-Information)

The amount of information gained by the occurrence of an event is inversely related to its probability.

Formula: If an event $x_i$ has probability $p_i$ :
$I(x_i) = \log_2\left(\frac{1}{p_i}\right) = -\log_2(p_i) \quad \text{bits}$
Properties:
- If $p_i = 1$ (certainty), $I = 0$ (no information gained).
- If $p_i \to 0$ (rare event), $I \to \infty$ (high information).

4.2 Entropy ( $H$ )

Entropy is the average information content per symbol of a source. It represents the uncertainty of the source.

Formula:
$H(X) = \sum_{i=1}^{M} p_i \log_2\left(\frac{1}{p_i}\right) = -\sum_{i=1}^{M} p_i \log_2(p_i) \quad \text{bits/symbol}$
Properties:
- $H(X) \ge 0$ .
- Max Entropy: Occurs when all symbols are equiprobable ( $p_i = 1/M$ ).
  $H_{max} = \log_2 M$

4.3 Information Rate ( $R$ )

The rate at which information is generated by the source.

R = r H(X) \quad \text{bits/second}

Where

r

is the signaling rate (symbols per second).

5. Source Coding

The objective of source coding is to represent the source output with the minimum number of bits (Data Compression).

Code Length ( $L$ ): The number of bits assigned to a symbol.
Average Code Length ( $L_{avg}$ ): $\sum p_i l_i$
Coding Efficiency ( $\eta$ ):
$\eta = \frac{H(X)}{L_{avg}}$
Redundancy: $1 - \eta$

5.1 Shannon-Fano Coding

A top-down approach to coding.

List source symbols in decreasing order of probability.
Partition the set into two subsets with approximately equal total probabilities.
Assign '0' to the upper set and '1' to the lower set.
Repeat until subsets contain only one symbol.

5.2 Huffman Coding

A bottom-up approach (Optimal prefix code).

List source symbols in decreasing order of probability.
Combine the two symbols with the lowest probabilities into a new composite symbol with probability equal to their sum.
Reorder the list.
Repeat until probability sums to 1.0.
Trace back the tree to assign bits (0 for upper branch, 1 for lower branch).
- Note: Huffman coding guarantees the lowest possible average code length for symbol-by-symbol coding.

6. Channel Capacity and Shannon-Hartley Law

6.1 Mutual Information $I(X;Y)$

Measures the amount of information the output $Y$ contains about the input $X$ .

Formula:
- $H(X)$ : Source Entropy (Uncertainty before reception).
- $H(X|Y)$ : Equivocation (Uncertainty remaining after reception).
Channel Capacity ( $C$ ): The maximum possible mutual information over all possible input probability distributions.
$C = \max_{p(x)} I(X;Y)$

6.2 Capacity of Discrete Channels

Lossless Channel: $H(X|Y) = 0$ . $C = H(X)$ .
Deterministic Channel: $H(Y|X) = 0$ . $C = H(Y)$ .
Binary Symmetric Channel (BSC):
With error probability $p$ :
$C = 1 - H_{bin}(p) = 1 - [-p\log_2 p - (1-p)\log_2(1-p)]$

6.3 Shannon-Hartley Law (Continuous Channel Capacity)

Defines the maximum data rate (capacity) that can be transmitted over a continuous analog channel with bandwidth $B$ affected by Additive White Gaussian Noise (AWGN).

Formula:

Where:
- $B$ : Channel Bandwidth (Hz)
- $S$ : Signal Power (Watts)
- $N$ : Noise Power (Watts) within bandwidth $B$ ( $N = N_0 B$ )
- $S/N$ : Signal-to-Noise Ratio (linear scale, not dB).

6.4 Trade-off between Bandwidth and SNR

The Shannon-Hartley law implies that bandwidth and signal power can be exchanged to maintain a constant channel capacity.

Bandwidth Limited: If $B$ is small, a very high SNR is required to achieve high capacity.
Power Limited: If SNR is low, capacity can be maintained by increasing bandwidth $B$ .
Limiting Case ( $B \to \infty$ ):
As bandwidth approaches infinity, capacity does not become infinite; it reaches a limit determined by the signal energy per bit ( $E_b$ ) and noise spectral density ( $N_0$ ).
$C_{limit} \approx 1.44 \frac{S}{N_0}$
The Shannon Limit: Reliable communication is impossible if $E_b/N_0$ is below $\ln 2$ (approx 0.693), or -1.6 dB.