Expectation and Covariance

Expectation

The expectation of a random variable $X$ is

$$E[X] = \int x \, p(x) \, dx, \qquad \text{in discrete } E[X] = \sum_{x \in S} x \, P(X = x)$$

E.g. $X \sim \mathrm{Exp}(\lambda)$,

$$E[X] = \int_0^\infty x \cdot \lambda e^{-\lambda x} \, dx = \lambda \cdot \frac{1}{\lambda^2} = \frac{1}{\lambda}$$

Expectation is linear,

$$E[aX + bY] = aE[X] + bE[Y]$$

LOTUS (law of the unconscious statistician) states

$$E[g(X)] = \int g(x) \, p(x) \, dx$$

e.g. $E[X^3] = \int x^3 p(x) \, dx$.

Variance

Variance is the measure of spread of an RV with a distribution,

$$\mathrm{Var}(X) = E\big[(X - E[X])^2\big] = E[X^2] - E[X]^2$$

Expectation with Respect to a Distribution

$E_{p(x)}[f(X, \ldots)]$ means take the expectation with respect to the distribution of $X$, that is $p(x)$. An equivalent form is $E_{X \sim p(x)}[f(X, \ldots)]$.

E.g. let $X \sim \mathrm{Unif}(0,1)$, $Y \sim \mathrm{Unif}(0,2)$, with $X, Y$ independent. Then

$$E[XY] = E_{X,Y}[XY] = E_{p(x,y)}[XY] = \int_0^2 \int_0^1 xy \cdot \frac{1}{2} \cdot \frac{1}{1} \, dx \, dy = \frac{1}{2}$$

But

$$E_X[XY] = \int_0^1 xY \cdot \frac{1}{1} \, dx = Y \int_0^1 x \, dx = \frac{Y}{2}$$

which is still a random variable.

Conditional Expectation

A conditional expectation $E[X | Y]$ is itself a random variable, a function of $Y$, with

$$E[X | Y = y] = \int x \, p(x | y) \, dx$$

E.g. for the counting heads example (from here),

$$\begin{aligned} E[X | Y = 0] &= 0 \cdot \tfrac{1}{2} + 1 \cdot \tfrac{1}{2} = \tfrac{1}{2} \\ E[X | Y = 1] &= 1 \cdot \tfrac{1}{2} + 2 \cdot \tfrac{1}{2} = \tfrac{3}{2} \end{aligned} \qquad \text{so} \qquad E[X | Y] = \begin{cases} 1/2 & \text{if } Y = 0, \text{ with probability } 1/2 \\ 3/2 & \text{if } Y = 1, \text{ with probability } 1/2 \end{cases}$$

Tower Law states

$$E[X] = E_Y\big[E_{X|Y}[X | Y]\big]$$

i.e. first take the expectation given $Y$, which is an RV as a function of $Y$, then take the expectation over $Y$ to get $E[X]$.

E.g. let $X \sim \mathrm{Unif}(0,1)$ and $Y | X \sim \mathcal{N}(X, 1)$. Then

$$E[Y] = E_X\big[E[Y | X]\big] = E_X[X] = \tfrac{1}{2}$$

Covariance and Correlation

A measure of relation or dependence between random variables is covariance,

$$\mathrm{Cov}(X, Y) = E\big[(X - E[X])(Y - E[Y])\big] = E[XY] - E[X]E[Y]$$

and $\mathrm{Cov}(X, X) = \mathrm{Var}(X)$.

Correlation is a measure of linear relation between two RVs,

$$\rho_{X,Y} = \frac{\mathrm{Cov}(X, Y)}{\sqrt{\mathrm{Var}(X)\, \mathrm{Var}(Y)}} \in [-1, 1]$$

$-1$ or $1$ means the RVs are linear transformations of each other, e.g. $Y = 3X + 2$.

If $X, Y$ are independent,

$$E[XY] = E[X]E[Y] \implies \mathrm{Cov}(X, Y) = 0 \implies \rho_{X,Y} = 0$$

But $\rho_{X,Y} = 0$ does not mean $X, Y$ are independent.

Random Vectors

Instead of 1D RVs, we can define random vectors with each entry being an RV. Then

$$X = \begin{bmatrix} X_1 \\ \vdots \\ X_n \end{bmatrix}, \qquad E[X] = \begin{bmatrix} E[X_1] \\ \vdots \\ E[X_n] \end{bmatrix}, \qquad \mathrm{Cov}(X) = \begin{bmatrix} \mathrm{Var}(X_1) & \cdots & \mathrm{Cov}(X_1, X_n) \\ \vdots & \ddots & \vdots \\ \mathrm{Cov}(X_n, X_1) & \cdots & \mathrm{Var}(X_n) \end{bmatrix} \in \mathbb{R}^{n \times n}$$

The covariance matrix $\Sigma = \mathrm{Cov}(X)$ is a symmetric, positive semidefinite matrix of the pairwise covariances.

Multivariate Normal

A multivariate normal is the distribution of a random vector where every linear combination of its entries is normally distributed. In particular each entry is normal, but normal entries alone are not enough.

It is specified as $X \sim \mathcal{N}(\mu, \Sigma)$, where $\mu$ is the vector of means and $\Sigma$ is the covariance matrix. If $\Sigma$ is invertible, for $n$ dimensional $X$

$$f_X(x) = \frac{1}{(2\pi)^{n/2} |\Sigma|^{1/2}} \exp\Big(-\frac{1}{2}(x - \mu)^T \Sigma^{-1} (x - \mu)\Big)$$

where $|\Sigma|$ is the determinant. Also,

$$AX + b \sim \mathcal{N}(A\mu + b,\ A\Sigma A^T)$$