ANOVA and Linear Regression

Setup

$I$ populations with the same variance and different means, $Y_{ij}$, $1 \le j \le n_i$, $1 \le i \le I$, independent with $Y_{ij} \sim \mathcal{N}(\mu_i, \sigma^2)$. With the common term $\mu = \bar{\mu} = \frac{1}{I}\sum_{i=1}^I \mu_i$ (non-random) and $\alpha_i = \mu_i - \mu$, $\sum_{i=1}^I \alpha_i = \sum_i \mu_i - I\mu = 0$, so

$$Y_{ij} \sim \mathcal{N}(\mu + \alpha_i, \sigma^2), \qquad Y_{ij} = \mu + \alpha_i + \varepsilon_{ij}, \quad \varepsilon_{ij} \sim \mathcal{N}(0, \sigma^2)$$ $$\bar{Y}_{i\cdot} = \frac{1}{n_i}\sum_{j=1}^{n_i} Y_{ij}, \qquad N = \sum_{i=1}^I n_i, \qquad \bar{Y}_{\cdot\cdot} = \frac{1}{N}\sum_{i=1}^I\sum_{j=1}^{n_i} Y_{ij} = \sum_{i=1}^I \frac{n_i}{N}\bar{Y}_{i\cdot}$$

so $\bar{Y}_{\cdot\cdot}$ is a weighted average of the group means, and the plain average $\frac{1}{I}\sum_i \bar{Y}_{i\cdot}$ when all $n_i = J$.

$$s_i^2 = \frac{1}{n_i - 1}\sum_{j=1}^{n_i}(Y_{ij} - \bar{Y}_{i\cdot})^2, \qquad \frac{(n_i - 1)s_i^2}{\sigma^2} \sim \chi^2_{n_i - 1}, \qquad \sum_{i=1}^I \frac{(n_i - 1)s_i^2}{\sigma^2} \sim \chi^2_{N - I}$$

The overall variability is $\frac{1}{N - 1}\sum_i\sum_j (Y_{ij} - \bar{Y}_{\cdot\cdot})^2$, and the pooled variance

$$s_p^2 = \frac{\sum_{i=1}^I (n_i - 1)s_i^2}{\sum_{i=1}^I (n_i - 1)} = \frac{\sum_{i=1}^I (n_i - 1)s_i^2}{N - I}, \qquad \frac{(N - I)s_p^2}{\sigma^2} \sim \chi^2_{N - I}$$

Two Groups

$H_0 : \mu_1 = \mu_2$ against $H_A : \mu_1 \neq \mu_2$, i.e. $H_0 : \alpha_i = 0$ for all $i$. The GLR test rejects for large $|T|$,

$$T = \frac{\bar{Y}_{1\cdot} - \bar{Y}_{2\cdot}}{\sqrt{s_p^2\big(\frac{1}{n_1} + \frac{1}{n_2}\big)}} \sim t_{n_1 + n_2 - 2} \text{ under } H_0, \qquad T^2 \sim F_{1, n_1 + n_2 - 2} = F_{I - 1, N - I}$$

One-Way ANOVA

Let all $n_i = J$, $H_0$: all means equal.

$$TSS = \sum_{i=1}^I\sum_{j=1}^J (Y_{ij} - \bar{Y}_{\cdot\cdot})^2, \qquad SS_W = SSE = \sum_{i=1}^I\sum_{j=1}^J (Y_{ij} - \bar{Y}_{i\cdot})^2 = \sum_{i=1}^I (J - 1)s_i^2, \qquad SS_B = SS_{Tr} = J\sum_{i=1}^I (\bar{Y}_{i\cdot} - \bar{Y}_{\cdot\cdot})^2$$

the total, within-sample, and between-sample (treatment) sums of squares, with $TSS = SS_W + SS_B$ and $s_p^2 = \frac{SS_W}{I(J - 1)}$.

$$\frac{SS_W}{\sigma^2} \sim \chi^2_{I(J - 1)}$$

Under $H_0$, $\bar{Y}_{i\cdot} \sim \mathcal{N}(\mu, \sigma^2/J)$, so $\frac{SS_B}{\sigma^2} \sim \chi^2_{I - 1}$, independent of $SS_W$, and

$$F = \frac{SS_B / (I - 1)}{SS_W / (I(J - 1))} = \frac{MS_{Tr}}{MS_E} \sim F_{I - 1, I(J - 1)}$$

Reject for large $F$; the p-value is $P(F_{I-1, I(J-1)} > F_{obs})$, and $H_0$ is rejected if it is less than $\alpha$. For independent $X_i \sim \mathcal{N}(\mu_i, \sigma^2)$, $E[(X_i - \bar{X})^2] = (\mu_i - \bar{\mu})^2 + \frac{n - 1}{n}\sigma^2$.

$$\begin{array}{l|l|l|l|l} \text{Source} & \text{Sum of squares} & \text{df} & \text{Mean square} & F \\ \hline \text{Between} & SS_{Tr} \; (SS_B) & I - 1 & MS_{Tr} = \frac{SS_{Tr}}{I - 1} & \frac{MS_{Tr}}{MS_E} \\ \text{Within} & SSE \; (SS_W) & I(J - 1) & MS_E = \frac{SSE}{I(J - 1)} & \\ \text{Total} & TSS & IJ - 1 & & \end{array}$$

Two-Way ANOVA

$Y_{ijk}$ is the $k$-th of $K$ replicates with factor A at level $i$ and factor B at level $j$.

$$\begin{array}{l|l|l} \text{Source} & \text{Sum of squares} & \text{df} \\ \hline \text{A} & SS_A & I - 1 \\ \text{B} & SS_B & J - 1 \\ \text{Interaction} & SS_{AB} & (I - 1)(J - 1) \\ \text{Error} & SSE & IJ(K - 1) \\ \text{Total} & TSS & IJK - 1 \end{array}$$

Simple Linear Regression

$Y = \beta x + a + \epsilon$, $\epsilon \sim \mathcal{N}(0, \sigma^2)$. Estimate $\hat{y} = \hat{\beta}x + \hat{a}$ by minimizing the squared error $\sum_{i=1}^n [y_i - (\beta x_i + a)]^2$,

$$\hat{\beta} = \frac{\sum_{i=1}^n (x_i - \bar{x})(y_i - \bar{y})}{\sum_{i=1}^n (x_i - \bar{x})^2} = \frac{S_{xy}}{S_{xx}}, \qquad \hat{a} = \bar{y} - \hat{\beta}\bar{x}$$

with $S_{xx} = \sum (x_i - \bar{x})^2$, $S_{xy} = \sum (x_i - \bar{x})(y_i - \bar{y})$, and

$$TSS = S_{yy} = \sum_{i=1}^n (y_i - \bar{y})^2 = \underbrace{\sum_{i=1}^n (y_i - \hat{y}_i)^2}_{SSE} + \underbrace{\sum_{i=1}^n (\hat{y}_i - \bar{y})^2}_{SSR}$$ $$MSE = \frac{SSE}{n - 2}, \qquad E[MSE] = \sigma^2, \qquad \mathrm{Var}(MSE) = \frac{\sigma^4}{(n - 2)/2}$$