Session 7: Eigenfaces

Faces as Matrices

Representing a Data Set of Faces as a Matrix

We will create a data set $F$ of gray-scale images. Each image will be a row in $F$, and their pixels will be placed in the columns.

The accompanying folder faces/ has a data set of 178 black-and-white face images. Out of these 178 faces, there are a few of them (indicated in face_descriptions.py) that should be omitted. Each image is of size $64\times 64$ pixels.

To build our face data set matrix $F$,

  1. Reshape each $i^{th}$ image into a row vector, $\mathbf{f}_i^T$, and
  2. Stack all of our $1\times 64^2$ vectors in $F$.

First, let's define some utility functions. These functions assume that images are gray-scale matrices of uint8 values between 0 and 255, and vectors are double precision so that we can perform arithmetics on them.

Building Matrix $F$

By using these functions we can build the $n$-by-$p$ matrix $F$ with $n = 169$ faces stacked as row vectors with $p = 64^2$ columns. This way, the image vector $\mathbf{f}_i^T$ is at the $i^{th}$ row of $F$:

$$ F = \begin{array}{r} \textrm{face image 0} \\ \textrm{face image 1} \\ \vdots \\ \textrm{face image }n - 1 \\ \end{array}\begin{pmatrix} f_{0,0} & f_{0,1} & \cdots & f_{0,p-1} \\ f_{1,0} & f_{1,1} & \cdots & f_{1,p-1} \\ \vdots & \vdots & \ddots & \vdots \\ f_{n-1,0} & f_{n-1,1} & \cdots & f_{n-1,p-1} \\ \end{pmatrix} \in \mathbb{R}^{169\times 64^2}. $$

Let's next load all images in the faces/ directory to build $F$.

The Average Face

We can compute the average face as $\mathbf{m} = \frac{1}{n}\sum_{i}\mathbf{f}_i$.

PCA of the Face Matrix

Let $\mathbf{m}^T$ be the mean row vector of matrix $F$ as defined above. Suppose we stack $\mathbf{m}^T$, yet again, $n$ times in a matrix $M$. Then, the covariance matrix of $F$ is given by

$$C =\textrm{cov}(F) = \frac{1}{n-1}(F - M)^T (F - M)$$

If we define $X = (F - M)$, then

$$\textrm{cov}(X) = \frac{1}{n-1}X^T X = \frac{1}{n-1}(F - M)^T (F - M) = \textrm{cov}(F) = C,$$

and

$$C = USV^T = VSV^T,$$

since $C$ is symmetric positive semidefinite and $U = V$. We call the columns in $V$ the principal components of $F$, and we can use them to express any face $\mathbf{f}$. Suppose $\mathbf{x} = \mathbf{f} - \mathbf{m}$ is a vectorized/flattened face image with all pixel values centered around the corresponding means. We can express $\mathbf{x}$ as

$$\mathbf{x} = V\mathbf{c} = c_0\mathbf{e}_0 + c_1\mathbf{e}_1 + \cdots + c_{p-1}\mathbf{e}_{p-1},$$

where $\mathbf{c} = (c_0, c_1, ..., c_{p-1})^T$ is a vector of coefficients, and

$$ V = \left(\begin{array}{c|c|c|c} ~ & ~ & ~ & ~ \\ \mathbf{e}_0 & \mathbf{e}_1 & \cdots & \mathbf{e}_{p-1} \\ ~ & ~ & ~ & ~ \\ \end{array}\right) $$

is the $p$-by-$p$ matrix of singular vectors of $\textrm{cov}(X)$.

To find the vector of coefficients/projections $\mathbf{c}$, we use

$$\mathbf{c} = V^T \mathbf{x},$$

and so $\mathbf{x} = V\mathbf{c} = V\left(V^T \mathbf{x}\right) = \mathbf{x}$. Furthermore, since $\mathbf{x} = \mathbf{f} - \mathbf{m}$, then

$$\mathbf{x} = (\mathbf{f} - \mathbf{m}) = V\mathbf{c},$$

which implies

$$\mathbf{f} = \mathbf{m} + V\mathbf{c}.$$

This is our cookbook to fabricate faces.

Eigenfaces

With PCA, we can perform data compression or low-rank approximation if we use only the first $k$ singular vectors in $V$. If we do so with any face $\mathbf{f}$, then

$$\mathbf{f}^{(k)} = \mathbf{m} + V^{(k)}\mathbf{c} = \mathbf{m} + c_0\mathbf{e}_0 + c_1\mathbf{e}_1 + \cdots + c_{k-1}\mathbf{e}_{k-1}.$$

The singular vectors in $V$ are called eigenfaces.

By modifying the coefficient of an eigenface (i.e., $c_j$ for $\mathbf{e}_j$ above), we can alter the appearance of the face. Let's modify the coefficient for the fourth (i.e., $j = 3$) eigenface, which emphasizes the tip of the nose and the eyebrows. In general, the brighter the pixels in an eigenface, the more it emphasizes those pixels/features in a face.

The $k^{th}$ Approximation of a Face Using Eigenfaces

By having our face cookbook ($V$ = eigenfaces) at hand, we can approximate a face $\mathbf{f}$ with just $k$ numbers!

$$\mathbf{f}^{(k)} = \mathbf{m} + V^{(k)}\mathbf{c} = \mathbf{m} + c_0\mathbf{e}_0 + c_1\mathbf{e}_1 + \cdots + c_{k-1}\mathbf{e}_{k-1}.$$

Generating Random Faces

By knowing the mean face $\mathbf{m}$ and the eigenfaces in $V$, we can synthesize any face $\tilde{\mathbf{f}}$ by sampling a vector of coefficients $\tilde{\mathbf{c}}$ from some normal distribution:

$$\tilde{\mathbf{f}}^{(k)} = \mathbf{m} + V^{(k)}\tilde{\mathbf{c}} = \mathbf{m} + \tilde{c}_0\mathbf{e}_0 + \tilde{c}_1\mathbf{e}_1 + \cdots + \tilde{c}_{k-1}\mathbf{e}_{k-1}.$$