Probability Spaces

Probability Space

A probability space is a triple $(\Omega, \mathcal{B}, P)$ where $\Omega$ is the sample space, $\mathcal{B}$ is a $\sigma$-field of events, and $P : \mathcal{B} \to [0,1]$ is a probability measure satisfying

1. $P(A) \ge 0$ for all $A \in \mathcal{B}$,
2. for pairwise disjoint $A_1, A_2, \ldots \in \mathcal{B}$, $P(\bigcup_{n=1}^{\infty} A_n) = \sum_{n=1}^{\infty} P(A_n)$,
3. $P(\Omega) = 1$.

Properties

$P(A^c) = 1 - P(A)$, $P(\emptyset) = 0$, and if $A \subseteq B$ then $P(A) \le P(B)$.

Inclusion-exclusion,

$$P\Big(\bigcup_{j=1}^{n} A_j\Big) = \sum_{r=1}^{n} (-1)^{r+1} \sum_{1 \le i_1 < \cdots < i_r \le n} P(A_{i_1} \cap \cdots \cap A_{i_r})$$

For any events,

$$P\Big(\bigcup_{n=1}^{\infty} A_n\Big) \le \sum_{n=1}^{\infty} P(A_n)$$

Continuity: if $A_n \uparrow A$ then $P(A_n) \uparrow P(A) = P(\bigcup_n A_n)$, and if $A_n \downarrow A$ then $P(A_n) \downarrow P(A) = P(\bigcap_n A_n)$.

Fatou's lemma for sets,

$$P(\liminf_{n \to \infty} A_n) \le \liminf_{n \to \infty} P(A_n) \le \limsup_{n \to \infty} P(A_n) \le P(\limsup_{n \to \infty} A_n)$$

Distribution Function

For $P$ on $(\mathbb{R}, \mathcal{B}(\mathbb{R}))$ define $F(x) = P((-\infty, x])$ for all $x$. Then

1. $F$ is right-continuous, $F(x+) = \lim_{y \downarrow x} F(y) = F(x)$,
2. $F$ is non-decreasing,
3. $\lim_{x \to -\infty} F(x) = 0$ and $\lim_{x \to \infty} F(x) = 1$.

A function $F : \mathbb{R} \to [0,1]$ satisfying the above is a distribution function (df).

Dynkin's $\pi$-$\lambda$ Theorem

Let $\mathcal{P}$ be a $\pi$-system.

1. If $\mathcal{L}$ is a $\lambda$-system and $\mathcal{P} \subseteq \mathcal{L}$, then $\sigma(\mathcal{P}) \subseteq \mathcal{L}$.
2. $\sigma(\mathcal{P}) = \mathcal{L}(\mathcal{P})$, the smallest $\lambda$-system containing $\mathcal{P}$.

A class that is both a $\pi$-system and a $\lambda$-system is a $\sigma$-field.

Uniqueness

For probability measures $P_1, P_2$ on $(\Omega, \mathcal{B})$,

$$\mathcal{L} = \{A \in \mathcal{B} : P_1(A) = P_2(A)\}$$

is a $\lambda$-system. So if $\mathcal{P} \subseteq \mathcal{B}$ is a $\pi$-system and $P_1(A) = P_2(A)$ for all $A \in \mathcal{P}$, then $P_1(B) = P_2(B)$ for all $B \in \sigma(\mathcal{P})$.

In particular, if $P_1, P_2$ have the same df, i.e. $P_1((-\infty, x]) = P_2((-\infty, x])$ for all $x \in \mathbb{R}$, then $P_1 = P_2$ on $\mathcal{B}(\mathbb{R})$.

Constructions

Discrete Associate $p_i$ to $\omega_i$ such that $p_i \ge 0$, $\sum_i p_i = 1$. Then $P(A) = \sum_{\omega_i \in A} p_i$.

General Specify $P(B)$ for $B$ in a class $\mathcal{S}$ of subsets of $\Omega$, then extend $P$ to $\sigma(\mathcal{S})$.

For $\mathcal{G}_1 \subseteq \mathcal{G}_2 \subseteq 2^\Omega$ and $P_i : \mathcal{G}_i \to [0,1]$, $P_2$ is an extension of $P_1$ if $P_2(B) = P_1(B)$ for $B \in \mathcal{G}_1$, equivalently $P_2|_{\mathcal{G}_1} = P_1$.

$P$ is finitely additive on $\mathcal{G}$ if for pairwise disjoint $A_1, \ldots, A_n \in \mathcal{G}$ with $\bigcup_{i=1}^{n} A_i \in \mathcal{G}$,

$$P\Big(\bigcup_{i=1}^{n} A_i\Big) = \sum_{i=1}^{n} P(A_i)$$

Half-open intervals in $\mathbb{R}$ and half-open rectangles in $\mathbb{R}^n$ form semialgebras. For a semialgebra $\mathcal{S}$,

$$\mathcal{A}(\mathcal{S}) = \Big\{\sum_{i \in I} S_i : I \text{ finite},\ \{S_i\}_{i \in I} \text{ disjoint},\ S_i \in \mathcal{S}\Big\}$$

is the algebra generated by $\mathcal{S}$, every set in it being a finite disjoint union of members of $\mathcal{S}$.

First Extension Let $\mathcal{S}$ be a semialgebra and $P : \mathcal{S} \to [0,1]$ $\sigma$-additive with $P(\Omega) = 1$. Then $P$ has a unique $\sigma$-additive extension $\tilde{P}$ to $\mathcal{A}(\mathcal{S})$. Specifically, if $A = \sum_{i \in I} B_i$ with $B_i \in \mathcal{S}$, then $\tilde{P}(A) = \sum_{i \in I} P(B_i)$.

Second Extension Let $\mathcal{A}$ be an algebra of subsets of $\Omega$ and $P : \mathcal{A} \to [0,1]$ $\sigma$-additive with $P(\Omega) = 1$. Then $P$ has a unique extension $\tilde{P}$ to a probability measure on $(\Omega, \sigma(\mathcal{A}))$.

Combining the two, a $\sigma$-additive $P : \mathcal{S} \to [0,1]$ on a semialgebra with $P(\Omega) = 1$ extends to a unique probability measure on $(\Omega, \sigma(\mathcal{S}))$,

$$\mathcal{S} \xrightarrow{\text{first ext}} \mathcal{A}(\mathcal{S}) \xrightarrow{\text{second ext}} \sigma(\mathcal{A}(\mathcal{S})) = \sigma(\mathcal{S})$$

Lebesgue Measure on (0,1]

$\lambda((a,b]) = b - a$ on $\{(a,b] : 0 \le a \le b \le 1\}$ is $\sigma$-additive, so it extends uniquely to $\mathcal{B}((0,1])$.

The $\sigma$-additivity uses compactness. $K \subseteq \mathbb{R}$ is compact if every open cover of $K$ has a finite subcover. Closed bounded intervals are compact.