An explanation and application of Principal component analysis (PCA)¶

PCA is a statistical technique for reducing the dimensionality of a dataset. This is accomplished by linearly transforming the data into a new coordinate system where (most of) the variation in the data can be described with fewer dimensions than the initial data.

• Jolliffe I. Principal Component Analysis (2ed., Springer, 2002)

Definition and Derivation of Principal Components¶

Definning PCs¶

Suppose that $ X $ is a vector of $p$ random variables, $$X= (x_1,\dots,x_p)$$, and that the variances of the $p$ random variables and the structure of the covariances or correlations between the $p$ variables are of interest. Unless $p$ is small, or the structure is very simple, it will often not be very helpful to simply look at the $p$ variances and all of the $\binom{p}{2} = \frac{p!}{2!(p-2)!}=p(p - 1)/2$ correlations or covariances. An alternative approach is to look for a few $(<<p)$ derived variables that pre- serve most of the information given by these variances and correlations or covariances.

Although PCA does not ignore covariances and correlations, it concentrates on variances. The first step is to look for a linear function $\alpha_1^T X$ of the elements of $X$ having maximum variance, where $\alpha_1$ is a vector of $p$ constants $\alpha_{11}, \alpha_{12}\dots,\alpha_{1p}$, and $T$ denotes transpose, so that

$$\alpha_1^T X=\alpha_{11} x_1 + \alpha_{12} x_2 +\cdots +\alpha_{1p} x_p =\sum_{j=1}^p\alpha_{1j} x_j $$

Next, look for a linear function $\alpha_2^T X$, uncorrelated with $\alpha_1^T X$ having maximum variance, and so on, so that at the $k$th stage a linear function $\alpha_k^T X$ is found that has maximum variance subject to being uncorrelated with $\alpha_1^T X, \alpha_2^T X,\dots, \alpha_{k-1}^T X$. The $k$th derived variable, $\alpha_k^T X$ is the $k$th PC. Up to $p$ PCs could be found, but it is hoped, in general, that most of the variation in $X$ will be accounted for by $m$ PCs, where $m <p$.

How to find them¶

Consider, for the moment, the case where the vector of random variables $X$ has a known covariance matrix $\Sigma$. This is the matrix whose $(i, j)$th element is the (known) covariance between the $i$th and $j$th elements of $X$ when $i \neq j$, and the variance of the $j$th element of $X$ when $i = j$. The more realistic case, where $\Sigma$ is unknown, follows by replacing $\Sigma$ by a sample covariance matrix $S$. It turns out that for $k = 1,2,··· ,p$, the $k$th PC is given by $z_k = \alpha_k^T X$ where $\alpha_k$ is an eigenvector of $\Sigma$ corresponding to its $k$th largest eigenvalue $\lambda_k$. Furthermore, if $\alpha_k$ is chosen to have unit length ($\alpha_k^T\alpha_k=1$), then $var(z_k) = \lambda_k$, where $var(z_k)$ denotes the variance of $z_k$.

Derive the form of the PCs¶

Consider first $\alpha_1^TX$; the vector $\alpha_1$ maximizes $var[\alpha_1^TX] = \alpha_1^T\Sigma \alpha_1$. It is clear that, as it stands, the maximum will not be achieved for finite $\alpha_1$ so a normalization constraint must be imposed. The constraint used in the derivation is $\alpha_1^T\alpha_1=1$, that is, the sum of squares of elements of $\alpha_1$ equals 1, ($\sum_{i=1}^p \alpha_{1i}^2=1$). Other constraints, for example $\text{Max}_j |\alpha_j | = 1$, may more useful in other circumstances, and can easily be substituted later on. However, the use of constraints other than $\alpha_1^T\alpha_1 = constant$ in the derivation leads to a more difficult optimization problem, and it will produce a set of derived variables different from the PCs.

A short review about variance of a random vector¶

• Definition 1: The expectation $E[X]$ of a random vector $X$ is given by $E[X]=(E[x_1],E[x_2],\cdots, E[x_p])^T$.
• Definition 2: The variance-covariance matrix (or simply the covariance

matrix ) of a random vector $X$ is given by $Cov (X) = E[(X- E[X])(X- E[X])^T]$

• Proposition 1: $Cov(X)=E[XX^T]-E[X]E[X]^T$
• Proposition 2: $Cov(X)=$

$$ \begin{bmatrix} Var(x_1) & Cov(x_1,x_2) & \cdots & Cov(x_1,x_p)\\ Cov(x_2) & Var(x_2,x_2) & \cdots & Cov(x_2,x_p)\\ \vdots&\vdots&\ddots&\vdots\\ Cov(x_p) & Cov(x_p,x_2) & \cdots & Var(x_p,x_p) \end{bmatrix} $$

Thus, $Cov(X)$ is symmetric matrix, since $Cov(X,Y)=Cov(Y,X)$.

• Proposition 3: Let $\alpha \in \mathbb{R}^p$, then $$Var(\alpha^T X)= \alpha^T Cov(X)\alpha$$

Maximizing $\alpha_1^T \Sigma \alpha_1$¶

Now, to maximize $\alpha_1^T \Sigma \alpha_1$ subject to $\alpha^T_1\alpha_1=1$, the standard approach is to use the technique of Lagrange multipliers. Maximize

$$\alpha^T_1\Sigma \alpha_1 - \lambda(\alpha_1^T \alpha_1 -1)$$

where $\lambda$ is a Lagrange multiplier. Differentiation with respect to $\alpha_1$ gives

$$\Sigma \alpha_1 - \lambda \alpha_1 = 0$$

, or $(\Sigma - \lambda I_p)\alpha_1=0$,

where $I_p$ is the ($p\times p $) identity matrix. Thus, $\lambda$ is an eigenvalue of $\Sigma$ and $\alpha_1$ is the corresponding eigenvector. To decide which of the p eigenvectors gives $\alpha_1^T X$ with maximum variance, note that the quantity to be maximized is

$\alpha_1^T\Sigma \alpha_1 = \alpha_1^T \lambda \alpha_1= \lambda \alpha_1^T \alpha_1 = \lambda$

so $\lambda$ must be as large as possible. Thus, $\alpha_1$ is the eigenvector corresponding to the largest eigenvalue of $\Sigma$, and $var(\alpha_1^T X) = \alpha_1^T \Sigma \alpha_1 = \lambda_1$, the largest eigenvalue.

In general, the $k$th PC of $X$ is $\alpha_k^TX$ and $var(\alpha_k^T) = \lambda_k$, where $\lambda_k$ is the $k$th largest eigenvalue of $\Sigma$, and $\alpha_k$ is the corresponding eigenvector. This will now be proved for $k = 2$; the proof for $k ≥ 3$ is slightly more complicated, but very similar.

Maximizing $\alpha_2^T \Sigma \alpha_2$¶

The second PC, $\alpha_2^T X$, maximizes $\alpha_2^T \Sigma \alpha_2$ subject to being uncorrelated with $\alpha_1^T X$, or equivalently subject to $cov[\alpha_1^T X,\alpha_2^T X] = 0$. But

$cov[\alpha_1^T X,\alpha_2^T X]=\alpha_1^T \Sigma \alpha_2 =\alpha_2^T \Sigma \alpha_1 = \alpha_2^T \lambda_1 \alpha_1^T = \lambda_1 \alpha_2^T \alpha_1 =\lambda_1 \alpha_1^T \alpha_2 $

Thus, any of the equations

$\alpha_1^T \Sigma \alpha_2 =0$, $\alpha_2^T \Sigma \alpha_1=0$, $\alpha_1^T\alpha_2 = 0$, $\alpha_2^T\alpha_1 = 0$

could be used to specify zero correlation between $\alpha_1^TX$ and $\alpha_2^TX$. Choosing the last of these (an arbitrary choice), and noting that a normalization constraint is again necessary, the quantity to be maximized is

$\alpha_2^T \Sigma \alpha_2 - \lambda(\alpha_2^T\alpha_2-1)-\phi\alpha_2^T\alpha_1$,

where $\lambda$, $\phi$ are Lagrange multipliers. Differentiation with respect to $\alpha_2$ gives

$\Sigma \alpha_2 -\lambda \alpha_2 -\phi\alpha_1 = 0$

and multiplication of this equation on the left by $\alpha_1^T$ gives

$\alpha_1^T\Sigma \alpha_2 -\lambda \alpha_1^T \alpha_2 -\phi\alpha_1^T\alpha_1 = 0$

which, since the first two terms are zero and $\alpha_1^T \alpha_1=1$, reduces to $\phi= 0$. Therefore, $\Sigma \alpha_2 -\lambda \alpha_2 = 0$, or equivalently $(\Sigma -\lambda I_p )\alpha_2 = 0$, so $\lambda$ is once more an eigenvalue of $\Sigma$, and $\alpha_2$ the corresponding eigenvector.

Again, $\lambda = \alpha_2^T \Sigma \alpha_2$, so $\lambda$ is to be as large as possible. Assuming that $\Sigma$ does not have repeated eigenvalues, $\lambda$ cannot equal $\lambda_1$. If it did, it follows that $\alpha_2 = \alpha_1$, violating the constraint $\alpha_1^T \alpha_2= 0$. Hence $\lambda$ is the second largest eigenvalue of $\Sigma$, and $\alpha_2$ is the corresponding eigenvector.

Now with $\alpha_k^T \Sigma \alpha_k$¶

As stated above, it can be shown that for the third, fourth, ..., $p$th PCs, the vectors of coefficients $\alpha_3, \alpha_4, . . . , \alpha_p$ are the eigenvectors of $\Sigma$ corresponding to $\lambda_3,\lambda_3,\dots,\lambda_p$, the third and fourth largest, ..., and the smallest eigenvalue, respectively. Furthermore,

$var[\alpha_k^T X] = \lambda_k$ for $k = 1,2,...,p.$

Application of PCA to a data set¶

• HOW DO YOU DO A PRINCIPAL COMPONENT ANALYSIS?

1. Standardize the range of continuous initial variables
2. Compute the covariance matrix to identify correlations
3. Compute the eigenvectors and eigenvalues of the covariance matrix to identify the principal components
4. Create a feature vector to decide which principal components to keep
5. Recast the data along the principal components axes

Exploration of the data¶

The data is in the archive gym_crowdedness.csv, it is the number of attendees every 10 minutes from one year in a gym.

The libraries and load the csv

https://www.kaggle.com/datasets/nsrose7224/crowdedness-at-the-campus-gym

We see a general glance at the data

Then we see if there is any null data element

In this example we will drop the column 'date' for simplicity

1. Standardize the range of continuous initial variables¶

2. Compute the covariance matrix to identify correlations¶

• If positive then: the two variables increase or decrease together (correlated)
• If negative then: one increases when the other decreases (Inversely correlated)

3. Compute the eigenvectors and eigenvalues of the covariance matrix to identify the principal components¶

4. Create a feature vector to decide which principal components to keep¶

From the previous graphic, we can select for example PCA_1, PCA2, PCA_6, PCA_7 and PCA_10 and we captured 78.79% of the variance.

Now we build the feature vector

5. Recast the data along the principal components axes¶

Created in Deepnote