Linear Algebra Review

Inner Product

$$\langle x, y \rangle = x \cdot y = x^T y = \sum_{i=1}^{n} x_i y_i$$

An inner product $\langle \cdot, \cdot \rangle$ is more general than the dot product. In $\mathbb{R}^n$, the dot product is usually used as the inner product as it induces the Euclidean norm. If $x \cdot y = 0$ then $x, y$ are orthogonal.

$$\|x\|_2 = \sqrt{\sum_{i=1}^{n} |x_i|^2} = \sqrt{\langle x, x \rangle} = \sqrt{x \cdot x} = \sqrt{x^T x}$$ $$\|x\|_1 = \sum_{i=1}^{n} |x_i|$$

$\|x\|$ usually means $\|x\|_2$.

The projection of $x$ on $y$ is

$$\frac{x \cdot y}{\|y\|^2} \, y$$

and the cosine similarity of $x$ and $y$ is

$$\frac{x \cdot y}{\|x\| \cdot \|y\|}$$

Square Matrices

For square matrices $A, B \in \mathbb{R}^{n \times n}$ and a scalar $c \in \mathbb{R}$, assume $A^{-1}, B^{-1}$ exist.

• $A$ is invertible $\iff$ $\det(A) \neq 0$ $\iff$ $A$ is full-rank $\iff$ columns of $A$ are linearly independent
• $(A^T)^{-1} = (A^{-1})^T$
• $(AB)^{-1} = B^{-1}A^{-1}$
• $\det(AB) = \det A \cdot \det B$
• $\det(A^{-1}) = \frac{1}{\det A}$
• $\det(cA) = c^n \det(A)$
• $A$ is orthogonal if $A^T A = I \iff A^{-1} = A^T$
• $\mathrm{tr}(A) = \sum_{i=1}^{n} A_{ii} = \sum_{i=1}^{n} \lambda_i$ for eigenvalues $\lambda_i$
• $A$ is symmetric if $A^T = A$

General Matrices

More generally, for matrices of compatible shapes,

• $(AB)^T = B^T A^T$
• $(A + B)^T = A^T + B^T$
• $\mathrm{tr}(AB) = \mathrm{tr}(BA)$ for $A \in \mathbb{R}^{n \times m}$, $B \in \mathbb{R}^{m \times n}$
• $AB \neq BA$ necessarily, matrix multiplication is not commutative

Positive and Negative Semi-Definiteness

For a symmetric $A \in \mathbb{R}^{n \times n}$,

• $A$ is positive semi-definite when $x^T A x \ge 0$ for all $x \in \mathbb{R}^n$ $\iff$ all eigenvalues of $A$ are $\ge 0$
• $A$ is negative semi-definite when $x^T A x \le 0$ for all $x \in \mathbb{R}^n$ $\iff$ all eigenvalues of $A$ are $\le 0$

Positive (negative) definite is the same with strict inequalities for $x \neq 0$.