Random Variables and Distributions

Random Variables and Distributions

A random variable $X$ is a map from the sample space $\Omega$ to a set $S$.

E.g. let $X$ be the RV that counts the number of heads for 2 coin flips. Then $\Omega = \{HH, HT, TH, TT\}$ and $S = \{0, 1, 2\}$.

A distribution of a random variable is a mapping from events (certain subset combinations of $\Omega$, this is a very non-rigorous definition) to $[0,1]$. E.g. for the example above the distribution is the collection of probabilities $P(X = x)$ for $x \in S$,

$$\begin{aligned} P(X = 0) &= P(\{TT\}) = \tfrac{1}{4} \\ P(X = 1) &= P(\{HT, TH\}) = \tfrac{1}{2} \\ P(X = 2) &= P(\{HH\}) = \tfrac{1}{4} \end{aligned}$$

For discrete cases the distribution is given via the probability mass function (PMF),

$$p_X(x) = P(X = x), \quad x \in S$$

remember $X$ is the RV, $x$ is a given value.

For continuous distributions we give the distribution with the probability density function (pdf), $p_X(x)$ or $f_X(x)$, defined by

$$P(a \le X \le b) = \int_a^b p_X(x) \, dx \quad \text{and} \quad \int_{\mathbb{R}} p_X(x) \, dx = 1$$

The cumulative distribution function (CDF) is

$$F_X(x) = P(X \le x)$$

So for any RV, $P(X = x)$ or $P(x_1 \le X \le x_2)$ is a specific case, not the distribution itself. The distribution is defined through a PMF $p_X(x)$ (discrete) or a CDF $F_X(x)$ (all RVs), which may define a pdf $p_X(x)$ (continuous).

We denote the distribution of an RV with the $\sim$ notation, i.e. $X \sim D(\theta)$ means $X$ is distributed with $D$ using parameters $\theta$. E.g. $X \sim \mathcal{N}(0,1)$ means $X$ is normally distributed with mean 0 and variance 1.

Joint Distributions

A joint distribution is the distribution induced by multiple RVs together. E.g. remember the distribution of $X$ defined $P(X = x)$ for all $x \in \{0,1,2\}$. Now let $Y$ be the RV that is 1 if the first flip is heads, 0 otherwise. Then the joint distribution of $X, Y$ is

$$\begin{aligned} P(X = 0, Y = 0) &= P(\{TT\}) = \tfrac{1}{4} & P(X = 0, Y = 1) &= P(\varnothing) = 0 \\ P(X = 1, Y = 0) &= P(\{TH\}) = \tfrac{1}{4} & P(X = 1, Y = 1) &= P(\{HT\}) = \tfrac{1}{4} \\ P(X = 2, Y = 0) &= P(\varnothing) = 0 & P(X = 2, Y = 1) &= P(\{HH\}) = \tfrac{1}{4} \end{aligned}$$

The pdf or pmf of the joint distribution of 2 RVs $X$ and $Y$ is denoted $p_{X,Y}(x,y)$, or simply $p(x,y)$. If discrete, $p_{X,Y}(x,y) = P(X = x, Y = y)$.

A marginal distribution is the distribution of only one RV in the context of a joint. E.g. $p_X(x), p_Y(y)$ are the marginals of $X$ and $Y$, $p_{X,Y}(x,y)$ is the joint distribution.

For any joint distribution, we can recover a marginal by integrating the other out,

$$p_X(x) = \int p_{X,Y}(x,y) \, dy, \qquad \text{collapses to} \qquad p_X(x) = \sum_{y} p_{X,Y}(x,y) \text{ in discrete}$$

If the joint distribution factorizes, the RVs are independent (marginally independent). If $p_{X,Y}(x,y) = p_X(x)\, p_Y(y)$ then $X$ and $Y$ are independent, and vice versa. Also denoted $X \perp\!\!\!\perp Y$.

Conditional Distributions

A conditional distribution is the distribution of one or multiple RVs conditioned on some other RVs. E.g. from the last example, in the conditional distribution of $X | Y$,

$$P(X = 0 \mid Y = 0) = \tfrac{1}{2}$$

as $Y = 0$ means $X$ can only be 0 or 1 with equal probability.

$p_{X|Y}(x|y)$, or $p(x|y)$, is used to denote the pdf or pmf of the conditional distribution. Again $p(x | Y = i)$ is conditioning on a specific realization of $Y$, while $p(x|y)$ is the whole conditional distribution.

$$p(x|y) = \frac{p(x,y)}{p(y)}$$

If $X$ and $Y$ are independent, $p(x|y) = p(x)$, as information on $Y$ doesn't give any information on $X$.

Note that abuse of notation is very common and not necessarily incorrect. For example, let $D$ be a random variable with $D = 1$ if someone has a disease and 0 if not. Then $P(\text{Sick}) = P(D = 1)$ would be an OK understanding without need for such a definition.

Law of Total Probability

The law of total probability states the probability of an event is the total probability of its different cases. For an event $A$,

$$p(A) = \int p(A | Y = y)\, p(y) \, dy \quad \text{or} \quad \sum_{y} p(A | Y = y)\, P(Y = y)$$

E.g. from the example,

$$P(X = 2) = P(X = 2 | Y = 0)P(Y = 0) + P(X = 2 | Y = 1)P(Y = 1) = 0 + \tfrac{1}{2} \cdot \tfrac{1}{2} = \tfrac{1}{4}$$

Conditional Independence

RVs $X$ and $Y$ are conditionally independent given an RV $Z$ if

1. $P(X | Y, Z) = P(X | Z)$,
2. $P(Y | X, Z) = P(Y | Z)$,
3. $P(X, Y | Z) = P(X | Z) \cdot P(Y | Z)$.

1 and 2 say knowledge of the other RV doesn't matter given $Z$, 3 says the joint distribution factorizes given $Z$. Also denoted $X \perp\!\!\!\perp Y \mid Z$.

This generalizes to more RVs, e.g.

$$X_1, X_2, X_3 \perp\!\!\!\perp Y_1, Y_2 \mid Z_1, Z_2 \implies P(X_1, X_2, X_3, Y_1, Y_2 | Z_1, Z_2) = P(X_1, X_2, X_3 | Z_1, Z_2)\, P(Y_1, Y_2 | Z_1, Z_2)$$

Semi-graphoid axioms For sets of RVs $X, Y, Z, W$,

1. $X \perp\!\!\!\perp Y \mid Z \iff Y \perp\!\!\!\perp X \mid Z$,
2. $X \perp\!\!\!\perp Y \cup W \mid Z \iff X \perp\!\!\!\perp W \mid Z$ and $X \perp\!\!\!\perp Y \mid Z \cup W$.

Bayes' Rule and Chain Rule

Bayes' Rule For RVs $A, B$,

$$P(A | B) = \frac{P(B | A)\, P(A)}{P(B)}$$

Chain Rule For RVs $X_1, \ldots, X_n$,

$$P(X_1, \ldots, X_n) = \prod_{i=1}^{n} P(X_i | X_{i-1}, \ldots, X_1)$$