Inner Product and Normed Spaces

Inner Product

For $V$ over $K$, an inner product is $\langle \cdot, \cdot \rangle : V \times V \to K$ with

1. $\langle x, x \rangle \in \mathbb{R}_{\ge 0}$, and $\langle x, x \rangle = 0$ iff $x = 0$,
2. $\langle x + y, z \rangle = \langle x, z \rangle + \langle y, z \rangle$,
3. $\langle cx, z \rangle = c \langle x, z \rangle$,
4. $\langle x, z \rangle = \overline{\langle z, x \rangle}$.

$(V, \langle \cdot, \cdot \rangle)$ is an inner product space.

Examples:

• $\mathbb{C}^n$ over $\mathbb{C}$, $\langle x, z \rangle = z^*x$,
• $\mathbb{C}^n$ over $\mathbb{C}$, $\langle x, z \rangle_A = z^*Ax$ for $A \in M_n$ Hermitian PD,
• $C^0[0,1]$ with a weight $w(t) > 0$, $\langle f, g \rangle = \int_0^1 f(t)\overline{g(t)}\, w(t)\, dt$.

In the second argument it is conjugate linear. For all $x, y, z, w \in V$ and $a, b, c, d \in K$,

$$\langle x, cz \rangle = \overline{\langle cz, x \rangle} = \overline{c \langle z, x \rangle} = \bar{c}\, \overline{\langle z, x \rangle} = \bar{c} \langle x, z \rangle$$ $$\langle x, y + z \rangle = \overline{\langle y + z, x \rangle} = \overline{\langle y, x \rangle} + \overline{\langle z, x \rangle} = \langle x, y \rangle + \langle x, z \rangle$$ $$\langle ax + by, cz + dw \rangle = a\bar{c} \langle x, z \rangle + a\bar{d} \langle x, w \rangle + b\bar{c} \langle y, z \rangle + b\bar{d} \langle y, w \rangle$$

Norm

For $V$ over $K$, a norm is a function $\| \cdot \| : V \to \mathbb{R}_{\ge 0}$ with

1. $\|x\| = 0$ iff $x = 0$,
2. $\|cx\| = |c| \, \|x\|$,
3. $\|x + y\| \le \|x\| + \|y\|$.

Examples: on $K^n$ over $K$, for $p \ge 1$,

$$\|x\|_p = \Big(\sum_i |x_i|^p\Big)^{1/p}, \qquad \|x\|_\infty = \max_i |x_i|$$

If $\|\cdot\|$ is a norm on $K^n$ and $A \in M_n(K)$ is invertible, then $\|x\|_A := \|Ax\|$ is a norm. The triangle inequality is

$$\|x + y\|_A = \|A(x + y)\| \le \|Ax\| + \|Ay\| = \|x\|_A + \|y\|_A$$

Cauchy-Schwarz

$$|\langle x, y \rangle|^2 \le \langle x, x \rangle \langle y, y \rangle$$

Proof For any $t, \theta \in \mathbb{R}$,

$$\begin{aligned} 0 &\le \langle te^{i\theta}x + y, te^{i\theta}x + y \rangle = t^2 e^{i\theta}\overline{e^{i\theta}} \langle x, x \rangle + te^{i\theta} \langle x, y \rangle + t\overline{e^{i\theta}} \langle y, x \rangle + \langle y, y \rangle \\ &= \langle x, x \rangle t^2 + 2\,\mathrm{Re}\{e^{i\theta} \langle x, y \rangle\} t + \langle y, y \rangle \end{aligned}$$

Choose $\theta$ such that $e^{i\theta} \langle x, y \rangle = |\langle x, y \rangle|$. Then $\langle x, x \rangle t^2 + 2|\langle x, y \rangle| t + \langle y, y \rangle \ge 0$ for all $t$, so the discriminant is

$$\Delta = (2|\langle x, y \rangle|)^2 - 4 \langle x, x \rangle \langle y, y \rangle \le 0$$

Induced Norm

If $V$ has an inner product, $\|x\| := \sqrt{\langle x, x \rangle}$ is a norm. The triangle inequality follows from Cauchy-Schwarz,

$$\|x + y\|^2 = \langle x, x \rangle + \langle x, y \rangle + \langle y, x \rangle + \langle y, y \rangle = \|x\|^2 + \|y\|^2 + 2\,\mathrm{Re}\langle x, y \rangle \le \|x\|^2 + \|y\|^2 + 2\|x\|\|y\| = (\|x\| + \|y\|)^2$$

Metric, Banach and Hilbert Spaces

$d : \mathcal{S} \times \mathcal{S} \to \mathbb{R}_{\ge 0}$ is a metric when

1. $d(x,y) = 0$ iff $x = y$,
2. $d(x,y) = d(y,x)$,
3. $d(x,z) \le d(x,y) + d(y,z)$,

and $(\mathcal{S}, d)$ is a metric space.

A Banach space is a normed linear space (NLS) that is complete in the metric of its norm. A Hilbert space is the same, with the norm created by an inner product. If $V$ is finite dimensional, any inner product or norm creates a complete space.

Finite Dimensional Normed Spaces

On an NLS $(V, \|\cdot\|)$, norms are continuous, since

$$\left. \begin{aligned} \|x\| &\le \|y\| + \|x - y\| \\ \|y\| &\le \|x\| + \|x - y\| \end{aligned} \right\} \implies \big|\|x\| - \|y\|\big| \le \|x - y\|$$

On an NLS $(V, \|\cdot\|)$ the following are equivalent

1. $V$ is finite dimensional,
2. the unit sphere $\{x \in V : \|x\| = 1\}$ is compact,
3. the unit ball $\{x \in V : \|x\| \le 1\}$ is compact,
4. for every $\omega \subseteq V$, $\omega$ is compact iff $\omega$ is closed and bounded.

All Norms are Equivalent If $V$ over $K$ is finite dimensional, for any norms $\|\cdot\|, \|\cdot\|'$ on $V$ there are $m, M \in \mathbb{R}_{>0}$ such that for all $x \in V$

$$m\|x\|' \le \|x\| \le M\|x\|'$$

Convergence

$x^{(i)} \to x$ in $\|\cdot\|$ means $\|x^{(i)} - x\| \to 0$. By equivalence of norms, $\|x^{(i)} - x\| \le M\|x^{(i)} - x\|'$, so if the right side goes to 0, so does the left. In finite dimensions, convergence doesn't depend on the norm.

Componentwise Convergence On $K^n$, $x^{(i)} \to x$ iff $x^{(i)}_j \to x_j$ for all $j$, since

$$\|x^{(i)} - x\| \to 0 \iff \|x^{(i)} - x\|_\infty \to 0 \iff |x^{(i)}_j - x_j| \to 0 \;\; \forall j$$

Linear Maps

For vector spaces $V, W$ over $K$, $T : V \to W$ is linear if for all $\alpha, \beta \in K$ and $x, y \in V$

$$T(\alpha x + \beta y) = \alpha T(x) + \beta T(y)$$

If $V$ is finite dimensional with $\dim V = n$, then $V$ is isomorphic to $K^n$ over $K$. Say $\mathcal{B} = \{b^{(1)}, \ldots, b^{(n)}\}$ is a basis. For all $x \in V$ there are unique $\alpha_1, \ldots, \alpha_n \in K$ with $x = \sum \alpha_i b^{(i)}$, and $x \mapsto (\alpha_1, \ldots, \alpha_n)$ is the isomorphism.

If $V, W$ are finite dimensional with bases $\mathcal{B}_V$ of size $n$ and $\mathcal{B}_W$ of size $m$, then $T : V \to W$ is linear iff there is $A \in M_{m,n}(K)$ such that for all $x \in V$, $y \in W$,

$$T(x) = y \iff A[x]_{\mathcal{B}_V} = [y]_{\mathcal{B}_W}$$