Expectation and Integration

Simple Random Variables

For a simple RV $X : \Omega \to \mathbb{R}$ with distinct values $a_1, \ldots, a_K$,

$$X = \sum_{i=1}^{K} a_i \mathbf{1}_{A_i}, \qquad A_i = \{X = a_i\}$$

The $A_i$ form a measurable partition of $\Omega$ and $\sigma(X) = \sigma(A_1, \ldots, A_K)$.

Let $\mathcal{E}$ be the collection of all measurable simple functions, and let $X = \sum_{i=1}^{K} a_i \mathbf{1}_{A_i}$, $Y = \sum_{j=1}^{m} b_j \mathbf{1}_{B_j}$.

1. $\mathcal{E}$ is a vector space over $\mathbb{R}$. For $c \in \mathbb{R}$, $cX = \sum_{i=1}^{K} c a_i \mathbf{1}_{A_i} \in \mathcal{E}$ and $X + Y = \sum_{i} \sum_{j} (a_i + b_j) \mathbf{1}_{A_i \cap B_j} \in \mathcal{E}$.
2. $\mathcal{E}$ is closed under products, max and min.

Measurability Theorem $X : \Omega \to [0, \infty]$ is measurable iff there is a sequence $X_n \in \mathcal{E}$ with $0 \le X_n \uparrow X$ pointwise. That is, $X \ge 0$ is measurable iff there are simple $X_n \uparrow X$.

Here $X_n \uparrow X$ means $X_n$ is non-decreasing and $X_n \to X$.

Expectation

For $X = \sum_{i=1}^{n} a_i \mathbf{1}_{A_i} \in \mathcal{E}$ with $a_i \in \mathbb{R}$, $A_i \in \mathcal{F}$,

$$E(X) = \int X \, dP = \sum_{i=1}^{n} a_i P(A_i)$$

On $\mathcal{E}$,

1. $E(1) = 1$, $E(\mathbf{1}_A) = P(A)$,
2. if $X \ge 0$ then $E(X) \ge 0$,
3. $E$ is linear,
4. $E$ is monotone,
5. if $X_n \uparrow X$ in $\mathcal{E}$ then $E(X_n) \uparrow E(X)$.

Let $\overline{\mathcal{E}}_+$ be the set of all nonnegative measurable RVs. For $X \in \overline{\mathcal{E}}_+$ choose $X_n \in \mathcal{E}$, $X_n \ge 0$, with $X_n \uparrow X$, and define

$$E(X) := \lim_{n \to \infty} E(X_n)$$

This does not depend on the choice of $X_n$. On $\overline{\mathcal{E}}_+$, $0 \le E(X) \le \infty$, and $E$ is monotone and linear. Convention: $0 \cdot \infty = 0$.

Monotone Convergence Theorem (MCT) If $X_n, X \in \overline{\mathcal{E}}_+$ and $X_n \uparrow X$, then $E(X_n) \uparrow E(X)$. Equivalently

$$E\Big(\lim_{n} X_n\Big) = \lim_{n} E(X_n)$$

As a consequence, if $\xi_j \ge 0$ are RVs,

$$E\Big(\sum_{j=1}^{\infty} \xi_j\Big) = \sum_{j=1}^{\infty} E(\xi_j)$$

Integrable Random Variables

For measurable $X : \Omega \to \overline{\mathbb{R}}$, let $X^+ := X \vee 0$ and $X^- := (-X) \vee 0$. Then $|X| = X^+ + X^-$ and $X = X^+ - X^-$.

$X$ is quasi-integrable if $E(X^+) < \infty$ or $E(X^-) < \infty$, and then

$$E(X) := E(X^+) - E(X^-)$$

$X$ is integrable if $E|X| = E(X^+) + E(X^-)$ is finite.

$$L^1(P) = \{X : \Omega \to \overline{\mathbb{R}} : X \text{ measurable},\ E|X| < \infty\}$$

1. If $X \in L^1(P)$ then $P(|X| = \infty) = 0$.
2. $E$ is linear on $L^1(P)$.
3. $E$ is monotone.
4. If $X_1 \in L^1(P)$ and $X_n \uparrow X$, then $E(X_n) \uparrow E(X)$.
5. $|E(X)| \le E|X|$.

$$\mathrm{Var}(X) := E[(X - E[X])^2] = E[X^2] - E[X]^2$$ $$\mathrm{Cov}(X,Y) := E[XY] - E[X]E[Y]$$

Fatou and Dominated Convergence

Fatou If $X_n \ge 0$ are RVs, then

$$E\Big[\liminf_{n \to \infty} X_n\Big] \le \liminf_{n \to \infty} E(X_n)$$

General Fatou If $X_n \ge Z$ with $Z \in L^1$, then $E[\liminf X_n] \le \liminf E(X_n)$.

Reverse Fatou If $X_n \le W$ with $W \in L^1$, then $\limsup E(X_n) \le E(\limsup X_n)$.

Dominated Convergence Theorem (DCT) If $X_n \to X$ a.s. and $|X_n| \le Z$ with $Z \in L^1$, then

$$E(X_n) \to E(X) \quad \text{and} \quad E|X_n - X| \to 0$$

Integrals over Sets

For $X \in L^1(P)$ and $A \in \mathcal{F}$,

$$\int_A X \, dP = E(X \mathbf{1}_A)$$

so $P(A) = E[\mathbf{1}_A]$. For measurable $X \ge 0$,

1. $0 \le \int_A X \, dP \le E(X)$,
2. $\int_A X \, dP = 0$ iff $P(A \cap \{X > 0\}) = 0$,
3. for pairwise disjoint $A_n$, $\int_{\cup A_n} X \, dP = \sum_{n=1}^{\infty} \int_{A_n} X \, dP$,
4. if $A_1 \subseteq A_2$, $\int_{A_1} X \, dP \le \int_{A_2} X \, dP$,
5. if $A_n \uparrow A$, $\int_{A_n} X \, dP \uparrow \int_A X \, dP$. If $A_n \downarrow A$ the same holds with $\downarrow$, but this needs $\int_{A_1} X \, dP < \infty$.

Change of Variables

For measurable $T : (\Omega, \mathcal{F}) \to (\Omega', \mathcal{F}')$, the induced measure is $P' = P \circ T^{-1}$, i.e. $P'(A') = P(T^{-1}(A'))$ for $A' \in \mathcal{F}'$. For measurable $X' : (\Omega', \mathcal{F}') \to (\mathbb{R}, \mathcal{B}(\mathbb{R}))$,

1. if $X' \ge 0$, $\int_\Omega X' \circ T \, dP = \int_{\Omega'} X' \, dP'$,
2. $X' \in L^1(P')$ iff $X' \circ T \in L^1(P)$, and then the same equality holds.

Let $X$ be a real valued RV with distribution $P_X = P \circ X^{-1}$ and $F(x) = P(X \le x)$. If $g : \mathbb{R} \to \mathbb{R}$ is Borel measurable and either $g \ge 0$ or $E|g(X)| < \infty$, then

$$E(g(X)) = \int_{\mathbb{R}} g(x) \, P_X(dx) = \int_{\mathbb{R}} g(x) \, dF(x)$$

In particular $E(X) = \int_{\mathbb{R}} x \, dF(x)$.

Densities

The distribution of $X \in \mathbb{R}^k$ is absolutely continuous with respect to Lebesgue measure if there is a nonnegative Borel measurable $f$ such that for all $A \in \mathcal{B}(\mathbb{R}^k)$,

$$P_X(A) = \int_A f(x_1, \ldots, x_k) \, dx_1 \cdots dx_k$$

Then $f \ge 0$ and $\int f = 1$.

Measures

A measure on $(\Omega, \mathcal{F})$ is $\mu : \mathcal{F} \to [0, \infty]$ such that $\mu(\emptyset) = 0$ and $\mu$ is $\sigma$-additive for disjoint sets in $\mathcal{F}$.

$\mu$ is $\sigma$-finite if there are $A_n \in \mathcal{F}$ with $\Omega = \bigcup_{n \ge 1} A_n$ and $\mu(A_n) < \infty$ for all $n$.

Riemann Integral

For bounded $f : [a,b] \to \mathbb{R}$ and a partition $Q : a = x_0 < x_1 < \cdots < x_n = b$, let $m_i := \inf_{x \in [x_{i-1}, x_i]} f(x)$ and $M_i := \sup_{x \in [x_{i-1}, x_i]} f(x)$.

$$L(f,Q) = \sum_{i=1}^{n} m_i (x_i - x_{i-1}), \qquad U(f,Q) = \sum_{i=1}^{n} M_i (x_i - x_{i-1})$$

$f$ is Riemann integrable if $\sup_Q L(f,Q) = \inf_Q U(f,Q)$. If $f : [a,b] \to \mathbb{R}$ is measurable and Riemann integrable, it is Lebesgue integrable with the same integral.

Almost Sure Equality

Let $X, X'$ be real RVs with $X \in L^1(P)$.

1. If $P(X = X') = 1$, then $X' \in L^1(P)$ and $E(X') = E(X)$.
2. Let $\mathcal{P} \subseteq \mathcal{F}$ be a $\pi$-system with $\Omega \in \mathcal{P}$ and $\sigma(\mathcal{P}) = \mathcal{F}$. If also $X' \in L^1(P)$, then $P(X = X') = 1$ iff $\int_A X \, dP = \int_A X' \, dP$ for all $A \in \mathcal{P}$.