Differential Entropy

For density $f$ with support $S$ ,

h(X) = -\int_S f(x) \log f(x)\, dx

It can be negative. For $X \sim \mathcal{N}(\mu, \sigma^2)$ ,

h(X) = \frac{1}{2}\ln(2\pi e\sigma^2) \text{ nats} = \frac{1}{2}\log(2\pi e\sigma^2) \text{ bits}

AEP For iid $X_1, \ldots, X_n$ , $-\frac{1}{n}\log f(X_1, \ldots, X_n) \to E[-\log f(X)] = h(X)$ in probability.

Quantization With $X^\Delta$ the quantization of $X$ to bins of width $\Delta$ , $H(X^\Delta) + \log \Delta \to h(X)$ as $\Delta \to 0$ .

Joint and Conditional

h(X_1, \ldots, X_n) = -\int f(x^n) \log f(x^n)\, dx^n, \qquad h(X|Y) = -\int f(x,y) \log f(x|y)\, dx\, dy = h(X,Y) - h(Y)

For $X \sim \mathcal{N}(\mu, K)$ in $\mathbb{R}^n$ ,

h(X) = \frac{1}{2}\log\big((2\pi e)^n |K|\big)

D(f \,\|\, g) = \int f \log \frac{f}{g}, \qquad I(X;Y) = \int f(x,y) \log \frac{f(x,y)}{f(x)f(y)}\, dx\, dy

$I(X^\Delta; Y^\Delta) \to I(X;Y)$ , and $I(X;Y) = \sup_{P,Q} I([X]_P; [Y]_Q)$ over finite partitions $P, Q$ .

For jointly Gaussian $X, Y$ with correlation $\rho = \sigma_{XY}/(\sigma_X \sigma_Y)$ ,

I(X;Y) = -\frac{1}{2}\log(1 - \rho^2)

$h(AX) = h(X) + \log |\det A|$ .
Maximum entropy If $X \in \mathbb{R}^n$ has zero mean and covariance $K = E[XX^T]$ , then $h(X) \le \frac{1}{2}\log\big((2\pi e)^n |K|\big)$ , with equality iff $X$ is Gaussian.
Estimation For any estimator $\hat{X}(Y)$ , with entropy in nats, $E\big[(X - \hat{X}(Y))^2\big] \ge \frac{1}{2\pi e} e^{2h(X|Y)}$ .