Parameter Estimation

Method of Moments

The $k$-th moment is $\mu_k = E[X^k]$ and the sample moment $\hat{\mu}_k = \frac{1}{n}\sum_{i=1}^n X_i^k$. Match them and solve for $\theta$.

The standard error $\sigma_{\hat{\theta}} = \sqrt{\mathrm{Var}(\hat{\theta})}$ depends on the true $\theta_0$, and $s_{\hat{\theta}}$ replaces $\theta_0$ with $\hat{\theta}$. E.g. for Poisson, $\hat{\lambda} = \bar{X}$, $\sigma_{\hat{\lambda}} = \sqrt{\lambda_0 / n}$, $s_{\hat{\lambda}} = \sqrt{\hat{\lambda} / n}$.

$\hat{\theta}_n$ is consistent if $P(|\hat{\theta}_n - \theta| > \epsilon) \to 0$ as $n \to \infty$ for all $\epsilon > 0$.

Maximum Likelihood

$$\mathrm{lik}(\theta) = f(x_1, \ldots, x_n | \theta) = \prod_{i=1}^n f(x_i | \theta) \text{ if IID}, \qquad l(\theta) = \sum_{i=1}^n \log f(x_i | \theta)$$

The Fisher information is

$$I(\theta) = E\Big[ \Big( \frac{\partial}{\partial \theta}\log f(X | \theta) \Big)^2 \Big] = -E\Big[ \frac{\partial^2}{\partial \theta^2}\log f(X | \theta) \Big]$$

(the second form if $f$ is smooth). Under regularity conditions (the support does not depend on the parameter, and $f$ is smooth in $\theta$):

• the MLE is consistent,
• $\sqrt{nI(\theta_0)}\,(\hat{\theta} - \theta_0) \to \mathcal{N}(0, 1)$ in distribution, so for large $n$ the MLE is approximately $\mathcal{N}\big(\theta_0, \frac{1}{nI(\theta_0)}\big)$, with asymptotic variance $\frac{1}{nI(\theta_0)} = \frac{1}{-E[l''(\theta_0)]}$,
• an approximate $100(1 - \alpha)\%$ confidence interval is $\hat{\theta} \pm z(\alpha/2) \frac{1}{\sqrt{nI(\hat{\theta})}}$, or $\hat{\theta} \pm z(\alpha/2)\, s_{\hat{\theta}}$.

Asymptotics

$$\sqrt{n}(\bar{X} - E[X_1]) \xrightarrow{d} \mathcal{N}(0, \sigma^2), \qquad \bar{X} \xrightarrow{P} E[X_1]$$

and for $g$ differentiable (delta method),

$$\sqrt{n}\big(g(\bar{X}) - g(E[X_1])\big) \xrightarrow{d} \mathcal{N}\big(0, g'(E[X_1])^2 \sigma^2\big)$$

Efficiency and the Cramér-Rao Bound

$$\mathrm{MSE} = \mathrm{Var}(\hat{\theta}) + \mathrm{Bias}^2 = \mathrm{Var}(\hat{\theta}) + (E[\hat{\theta}] - \theta_0)^2, \qquad \mathrm{eff}(\hat{\theta}, \tilde{\theta}) = \frac{\mathrm{Var}(\tilde{\theta})}{\mathrm{Var}(\hat{\theta})}$$

The efficiency is the ratio of sample sizes needed to get the same variance.

Cramér-Rao If $X_1, \ldots, X_n$ are IID with $f$ smooth and $T = t(X_1, \ldots, X_n)$ is unbiased,

$$\mathrm{Var}(T) \ge \frac{1}{nI(\theta)}$$

and without the unbiasedness,

$$\mathrm{Var}(T) \ge \frac{\big( \frac{\partial}{\partial \theta}E[T] \big)^2}{nI(\theta)}$$

Sufficiency

$T$ is sufficient for $\theta$ if the conditional distribution of $X_1, \ldots, X_n$ given $T$ does not depend on $\theta$.

Factorization $T$ is sufficient iff $f(x_1, \ldots, x_n | \theta) = g(T, \theta)\, h(x_1, \ldots, x_n)$. If $T$ is sufficient and $g$ is one-to-one, $g(T)$ is sufficient.

E.g. for $X_i \sim \mathcal{N}(\mu, \sigma^2)$,

$$f(x_1, \ldots, x_n | \mu, \sigma^2) = \Big( \frac{1}{\sigma\sqrt{2\pi}} \Big)^n \exp\Big( -\frac{1}{2\sigma^2}\sum_{i=1}^n (x_i - \mu)^2 \Big) = \Big( \frac{1}{\sigma\sqrt{2\pi}} \Big)^n \exp\Big( -\frac{1}{2\sigma^2}\Big[ \sum_{i=1}^n x_i^2 - 2\mu n\bar{x} + n\mu^2 \Big] \Big)$$

which depends on the data only through $\sum x_i^2$ and $\bar{x}$, so $(\bar{X}, S^2)$ is sufficient for $(\mu, \sigma^2)$.

Bayesian Estimation

The prior $f_\Theta(\theta)$ is the distribution of the unknown $\theta$, $f_{X|\Theta}(x | \theta)$ the likelihood, and the posterior is

$$f_{\Theta|X}(\theta | x) = \frac{f_{X,\Theta}(x, \theta)}{f_X(x)} = \frac{f_{X|\Theta}(x | \theta)\, f_\Theta(\theta)}{\int f_{X|\Theta}(x | \theta)\, f_\Theta(\theta)\, d\theta}$$

with the denominator constant in $\theta$. The posterior mean is a function of the observed data.

A loss function $l(\theta, a)$ measures the cost of estimating $\theta$ by $a$, e.g. squared error $(a - \theta)^2$, absolute error $|\theta - a|$, or 0-1 loss (0 if $a = \theta$, 1 otherwise). A decision rule $\delta$ maps the data to an estimate, e.g. $\delta(X_1, \ldots, X_n) = X_1$ or $\bar{X}$, and the goal is to minimize $E[l(\theta, \delta(X_1, \ldots, X_n))]$.