Channel Capacity

Discrete Memoryless Channels

A channel with input alphabet $\mathcal{X}$, output alphabet $\mathcal{Y}$, and transition probabilities $p(y|x)$ has capacity

$$C = \max_{p(x)} I(X;Y)$$

• $0 \le C \le \min(\log|\mathcal{X}|, \log|\mathcal{Y}|)$.
• $I(X;Y)$ is a continuous, concave function of $p(x)$.

Gaussian Channel

$$Y_i = X_i + Z_i, \qquad Z_i \sim \mathcal{N}(0, N)$$

with power constraint $\frac{1}{n}\sum_{i=1}^{n} x_i^2 \le P$ on each codeword. The capacity is

$$C = \max_{f(x):\, E[X^2] \le P} I(X;Y)$$

$I(X;Y) = h(Y) - h(Y|X) = h(Y) - h(Z)$, and with $X, Z$ independent,

$$E[Y^2] = E[X^2] + 2E[X]E[Z] + E[Z^2] \le P + N$$

maximized for $E[X^2] = P$. So

$$I(X;Y) \le \frac{1}{2}\log\big(2\pi e(P + N)\big) - \frac{1}{2}\log(2\pi e N) = \frac{1}{2}\log\left(1 + \frac{P}{N}\right)$$

with equality for $X \sim \mathcal{N}(0, P)$. So

$$C = \frac{1}{2}\log\left(1 + \frac{P}{N}\right) \text{ bits per transmission}$$

Band-Limited Channels

Nyquist-Shannon A function band-limited to $W$ Hz loses no information when sampled at most $\frac{1}{2W}$ seconds apart.

With noise spectral density $S_N(f) = N_0/2$, the capacity of the band-limited channel is

$$C = \frac{1}{2}\log\left(1 + \frac{P}{N_0 W}\right) \text{ bits per transmission}$$

and with unlimited bandwidth $C = \frac{P}{N_0}\log e$ bits per second.