Transformations and Order Statistics

Change of Variables

For $U = g_1(X, Y)$, $V = g_2(X, Y)$ one-to-one, with inverse $x = h_1(u, v)$, $y = h_2(u, v)$,

$$f_{U,V}(u, v) = f_{X,Y}\big(h_1(u, v), h_2(u, v)\big) \cdot |J|, \qquad J = \det \begin{bmatrix} \frac{\partial x}{\partial u} & \frac{\partial x}{\partial v} \\ \frac{\partial y}{\partial u} & \frac{\partial y}{\partial v} \end{bmatrix}$$

Order Statistics

For $X_1, \ldots, X_n$ IID with CDF $F$ and density $f$, and $X_{(1)} \le \cdots \le X_{(n)}$ the sorted values,

$$F_{X_{(n)}}(x) = F(x)^n, \qquad F_{X_{(1)}}(x) = 1 - [1 - F(x)]^n$$ $$f_{X_{(n)}}(x) = nF(x)^{n-1}f(x), \qquad f_{X_{(1)}}(x) = n[1 - F(x)]^{n-1}f(x)$$ $$f_{X_{(i)}}(x) = \binom{n}{i - 1, 1, n - i} F(x)^{i-1} f(x) [1 - F(x)]^{n-i}$$

For $i < j$,

$$f_{X_{(i)}, X_{(j)}}(x_i, x_j) = \binom{n}{i - 1, 1, j - i - 1, 1, n - j} F(x_i)^{i-1} f(x_i) [F(x_j) - F(x_i)]^{j-i-1} f(x_j) [1 - F(x_j)]^{n-j} \, I_{x_i < x_j}$$

and the joint density of all of them is

$$f(x_1, \ldots, x_n) = n! \, f(x_1) f(x_2) \cdots f(x_n) \, I_{x_1 < x_2 < \cdots < x_n}$$

Exchangeability

$X_1, \ldots, X_n$ are exchangeable if their joint distribution is the same under every permutation of $(1, \ldots, n)$, i.e. no variable is any more likely to be at a certain place. IID implies exchangeable. Sampling without replacement from a finite population gives exchangeable but not independent samples (see survey sampling).

Bivariate Normal

With $Z_1, Z_2$ IID $\mathcal{N}(0, 1)$,

$$X = \mu_1 + \sigma_1 Z_1, \qquad Y = \mu_2 + \sigma_2\big(\rho Z_1 + \sqrt{1 - \rho^2}\, Z_2\big)$$

gives $(X, Y) \sim \mathrm{BVN}(\mu_1, \mu_2, \sigma_1^2, \sigma_2^2, \rho)$, with $\rho(X, Y) = \frac{\mathrm{Cov}(X, Y)}{\sigma_X \sigma_Y} = \rho$.

Variance and Conditioning

$$\mathrm{Var}(X) = E[\mathrm{Var}(X | Y)] + \mathrm{Var}(E[X | Y])$$

By Cauchy-Schwarz $|E[XY]| \le (E[X^2]E[Y^2])^{1/2}$, so $|\rho(X, Y)| \le 1$, and $|\rho(X, Y)| = 1$ implies $Y = aX + b$.