Multivariable Calculus

Gradient and Hessian

For $f : \mathbb{R}^n \to \mathbb{R}$,

$$\nabla_x f = \begin{bmatrix} \frac{\partial f}{\partial x_1} \\ \vdots \\ \frac{\partial f}{\partial x_n} \end{bmatrix} \in \mathbb{R}^n, \qquad \nabla_x^2 f = \begin{bmatrix} \frac{\partial^2 f}{\partial x_1^2} & \cdots & \frac{\partial^2 f}{\partial x_1 \partial x_n} \\ \vdots & \ddots & \vdots \\ \frac{\partial^2 f}{\partial x_n \partial x_1} & \cdots & \frac{\partial^2 f}{\partial x_n^2} \end{bmatrix} \in \mathbb{R}^{n \times n}$$

Jacobian

For $g : \mathbb{R}^n \to \mathbb{R}^m$,

$$g(x_1, \ldots, x_n) = \begin{bmatrix} g_1(x_1, \ldots, x_n) \\ \vdots \\ g_m(x_1, \ldots, x_n) \end{bmatrix}$$

for each $g_i : \mathbb{R}^n \to \mathbb{R}$. Then the analogue of a derivative is the Jacobian matrix

$$J_g(x) = \nabla_x g(x) = \begin{bmatrix} \nabla g_1(x)^T \\ \vdots \\ \nabla g_m(x)^T \end{bmatrix} = \begin{bmatrix} \frac{\partial g_1}{\partial x_1} & \cdots & \frac{\partial g_1}{\partial x_n} \\ \vdots & \ddots & \vdots \\ \frac{\partial g_m}{\partial x_1} & \cdots & \frac{\partial g_m}{\partial x_n} \end{bmatrix} \in \mathbb{R}^{m \times n}$$

Integrals

The standard notation for an integral over $\vec{x} \in \mathbb{R}^n$ is

$$\int f(\ldots) \, d\vec{x} = \int \int \cdots \int f(\ldots) \, dx_1 \, dx_2 \ldots dx_n$$

Supremum and Infimum

For $f : V \to \mathbb{F}$ with an ordering on $\mathbb{F}$, $\sup f(x) = x_{\sup}$ such that

• $x_{\sup} \ge f(x)$ for all $x \in V$, i.e. $x_{\sup}$ is an upper bound,
• $x_{\sup} \le u$ for every upper bound $u$.

Note $x_{\sup} \in \mathbb{F}$ but it is not necessarily in the range of $f$. If $f$ has a max, it is the supremum. In $\mathbb{R}$, this is the concept of least upper bound.

And $\inf f(x) = x_{\inf}$ such that $x_{\inf} \le f(x)$ for all $x$, and $x_{\inf} \ge \ell$ for every lower bound $\ell$. In $\mathbb{R}$ this is the greatest lower bound. If a min exists, it is the infimum.

Taylor Expansions

For $f : \mathbb{R}^n \to \mathbb{R}$, we can approximate around a point $x_0$ using its Taylor series.

First-order approximation:

$$f(x) \approx f(x_0) + \nabla_x f(x_0)^T (x - x_0)$$

Second-order approximation:

$$f(x) \approx f(x_0) + \nabla_x f(x_0)^T (x - x_0) + \frac{1}{2}(x - x_0)^T \nabla^2 f(x_0)(x - x_0)$$

The notation $O(h^n)$ is used to account for the dropped terms in the infinite Taylor series. E.g. let $h = x - x_0$,

$$f(x_0 + h) = f(x_0) + \nabla f(x_0)^T h + O(\|h\|^2)$$

the error term (dropped terms) shrinks as fast as $\|h\|^2$.

Calculus with Matrices

Linear form For $a, x \in \mathbb{R}^n$, $f(x) = a^T x$,

$$\nabla_x f = a, \qquad \nabla_x^2 f = 0, \qquad \nabla_x(x) = I_n$$

Quadratic form For $x \in \mathbb{R}^n$, $A \in \mathbb{R}^{n \times n}$, $f(x) = x^T A x$,

$$\nabla_x f = (A + A^T)x, \qquad \nabla_x^2 f = A + A^T$$

Euclidean norm $f(x) = \|x\|_2^2 = x^T x$,

$$\nabla_x f = 2x, \qquad \nabla_x^2 f = 2I_n$$

Least squares For $A \in \mathbb{R}^{m \times n}$, $x \in \mathbb{R}^n$, $b \in \mathbb{R}^m$,

$$f(x) = \|Ax - b\|_2^2 = (Ax - b)^T(Ax - b) = x^T A^T A x - 2b^T A x + b^T b$$ $$\nabla_x f = 2A^T(Ax - b), \qquad \nabla_x^2 f = 2A^T A$$

Convexity

A function $f : \mathbb{R}^n \to \mathbb{R}$ is convex if

$$f(\lambda x + (1 - \lambda)y) \le \lambda f(x) + (1 - \lambda) f(y), \quad \forall x, y \in \mathbb{R}^n,\ \forall \lambda \in [0,1]$$

Intuitively, the line segment between any 2 points on the graph of $f$ lies above or on the graph. Concavity is the reverse, where the segment lies below.

For twice differentiable $f$, an equivalent statement is

• $f$ is convex iff $\nabla^2 f$ is positive semidefinite everywhere,
• $f$ is concave iff $\nabla^2 f$ is negative semidefinite everywhere.