Statistical Estimation

Parametric Models

A set of distributions indexed by a parameter $\theta$ is a parametric model. Note $\theta$ is used to denote any number of parameters, and $;$ is used to separate the parameters. E.g. $P(X, Y; \theta)$ is the joint distribution of $X$ and $Y$ using $\theta$ as the parameter. For $X \sim \mathrm{Exp}(\theta)$, the parametric family is

$$p(x; \theta) = \theta e^{-\theta x}, \quad x \ge 0$$

IID (or i.i.d.) means independent and identically distributed.

Probabilistic View of Data

The data $D$ can be seen as IID random variables with outputs $Y$. For $n$ data points with $k$ features each,

$$D = (X, Y) = \begin{bmatrix} X_{11} & \cdots & X_{1k} & Y_1 \\ \vdots & & \vdots & \vdots \\ X_{n1} & \cdots & X_{nk} & Y_n \end{bmatrix}$$

each row is a data point in $\mathbb{R}^k$ with its output $Y$. If we assume $P(X, Y)$ is parametric, the goal is to estimate the parameter $\theta$ in $P(X, Y; \theta)$.

Estimators

The estimator is usually notated with a hat, e.g. $\hat{\theta}$, and is a function of the data as random variables, i.e. $\hat{\theta} = g(X, Y)$. When the data is realized (we plug in sampled numbers, not RVs in the theoretical sense), the estimator is realized as the estimate.

E.g. let 3 data points be IID, $X_i \sim \mathcal{N}(\mu, \sigma^2)$, and we want to estimate $\mu$. A reasonable choice is

$$\hat{\mu} = \frac{1}{3} \sum_{i=1}^{3} X_i$$

This estimator is an RV we can analyze. Now imagine we collect 3 data points $90, 120, 150$. We can use the estimator above to get a sample mean

$$\hat{\mu} = \frac{90 + 120 + 150}{3} = 120$$

Bias, Variance and MSE

$$\mathrm{Bias}(\hat{\theta}) = E[\hat{\theta}] - \theta$$

The variance of an estimator is $\mathrm{Var}(\hat{\theta})$, and its mean-squared error is

$$\mathrm{MSE}(\hat{\theta}) = E\big[(\hat{\theta} - \theta)^2\big] = \mathrm{Var}(\hat{\theta}) + \mathrm{Bias}(\hat{\theta})^2$$

Frequentist Framework

In the frequentist view, the parameter $\theta$ is an unknown constant to be estimated from data. E.g. to estimate the fairness of a coin, let a flip be $\sim \mathrm{Bern}(\theta)$ and estimate $\theta$.

For data $x_1, \ldots, x_n$, the likelihood is the probability (or density) of the data under $\theta$,

$$\mathcal{L}(\theta) = \prod_{i=1}^{n} p(x_i; \theta)$$

The maximum likelihood estimator is the constant $\theta$ that maximizes the likelihood of the data,

$$\hat{\theta}_{\mathrm{MLE}} = \mathrm{argmax}_\theta \, \mathcal{L}(\theta)$$

The log-likelihood is $\ell(\theta) = \log \mathcal{L}(\theta)$. Since $\log$ is increasing, it has the same maximizer, and turns the product into a sum.

Bayesian Framework

In the Bayesian setting we are naturally uncertain about $\theta$, so $\theta$ itself is an RV, usually with a parametric distribution called the prior. E.g. in the coin flipping case, $X \sim \mathrm{Bern}(\theta)$, $\theta \sim \mathrm{Beta}(\alpha, \beta)$, and we update the distribution of $\theta$ given new data.

Using Bayes' Rule to update the distribution of $\theta$ given data $X$,

$$\underbrace{P(\theta | X)}_{\text{posterior}} = \frac{\overbrace{P(X | \theta)}^{\text{likelihood}} \, \overbrace{P(\theta)}^{\text{prior}}}{\underbrace{P(X)}_{\text{evidence}}} \propto P(X | \theta) P(\theta)$$

Since $P(X)$ is a constant with respect to $\theta$, $P(X | \theta) P(\theta)$ is enough to express the distribution. The constant is the one that makes $P(\theta | X)$ integrate to 1.

Then, the maximum a posteriori estimate is

$$\hat{\theta}_{\mathrm{MAP}} = \mathrm{argmax}_\theta \, P(\theta | X)$$