Principal Component Analysis and Loadings¶

Principal Component Analysis (PCA) is one of the most popular data analysis and dimensionality techniques to this day. As someone coming from ML background, I have seen it used for dimensionality reduction but for some reason it seems I never noticed the “analysis” in the name that much. Then, when I was reading a paper, I saw they used PCA to justify their groupings and analyze the data, and they did that by something called loadings. After a little bit of research, it turns out it is just a fancy name but I wanted to write this post as my first “official” blog post on something.

I will derive PCA from the maximum variance point of view, and then explain what loadings are and how they explain the relationship between features. Then, I will apply a PCA analysis on air quality measurements in Mersin, Turkey and we will see if we can capture similar stations by those loadings.

Let’s start.

First, let’s derive PCA ourselves, from a maximum variance point of view.

We are interested in the dataset $X \in \mathbb{R}^{n\times d}$, where each row represents a datapoint and each column represents a feature.

Before going further, let us normalize our data to have a mean of 0.

$$\begin{aligned} X_c = X – \bar{X} \end{aligned}$$

where $\bar{X}$ represents the row-wise means.

This is our data represented in the standard basis, i.e $\begin{pmatrix}1&0&0&0&0\end{pmatrix}$, $\begin{pmatrix}0&1&0&0&0\end{pmatrix}$, $\dots$

We are interested in other basis vectors, such that when projected using those vectors, variation is maximized. Let’s check the current variance first:

$$\begin{aligned} S &= \frac{1}{N} X_c^TX_c \end{aligned}$$

This is our current covariance matrix. When we project our data to a new vector, our new covariance matrix will be:

$$\begin{aligned} \frac{1}{N}(X_c u_1)^T(X_c u_1) &= \frac{1}{N} u_1^TX_c X_c u_1 = u_1^TSu_1 \end{aligned}$$

We need to maximize this variance. It is obvious that when $u_1 \to \infty$, this will be maximized. But we are interested in maximizing by finding a direction and then projecting onto it. Therefore, the magnitude of the vector is irrelevant. To make our life easier, we can fix its size to 1: $u_1^Tu_1 = 1$ and this would turn our problem into a constrained optimization problem.

We can introduce a Lagrange multiplier $\lambda_1$ and optimize the objective:

$$\begin{aligned} & u_1^TSu_1 + \lambda_1(1 – u_1^Tu_1) & \\ & 2 S u_1 – \lambda_1 u_1 = 0 & \text{(Take the derivative wrt. $u_1$ and set it to 0)}\\ & Su_1 = \lambda_1 u_1 \end{aligned}$$

From the final equation, we can see that $u_1$ needs to be an eigenvector of the covariance matrix $S$. Therefore, maximizing the variation would correspond to selecting the largest eigenvalue. In that case, $u_1$ is the first principal component (Or depending on your field, you might see it called a principal axis but in ML community, it is usually the principal component). Then, following this in an iterative way we can get all of our principal components.

But, how do we calculate the PCA? The answer is Singular Value Decomposition (SVD), as it is a highly related concept. All matrices have SVD, thus we know that our dataset matrix also has an SVD: $X_c = U\Sigma V^T$. Using this relationship, we can find the eigenvectors and eigenvalues of $S$:

$$\begin{align} X_c^T X_c &= V \Sigma U^T U \Sigma V^T = V \Sigma^2 V^T & \text{($U$ is a unitary matrix, s.t. $U^TU = I$)}\\ \implies & S = \frac{1}{N} V \Sigma^2 V^T \end{align}$$

And as we do not care about the magnitude of the eigenvectors (i.e. If $v_1$ is an eigenvector, $5v_1$ is also a valid eigenvector), we can just drop the $\frac{1}{N}$ and use $V \Sigma^2 V^T$. From here, we see two things:

1. Right singular vectors of the dataset matrix are scaled eigenvectors of the covariance matrix.
2. Singular values of the dataset matrix are the squareroots of the eigenvalues of the covariance matrix.

Then, we can simply calculate the PCA by using SVD:

Recovering the exact eigenvalues:

Each row is an eigenvector:

Now, we have the corresponding eigenvalues and eigenvectors, i.e. variances and principal components. We can project our data to each of these vectors:

And for dimensionality reduction, we can decide to just keep top 2:

Loadings¶

In some texts, it is possible to see something called “loadings”, which are used to understand the relationship between original features and the PCA transformation. It sounds fancy, but it is just the components of the eigenvectors. They call this is the contribution of the original feature to the eigenvector.

Here is the idea behind it: Let’s actually think about projecting. Let us consider the first principal component:

And consider our first data point:

To project $x_1$ to $u_1$, we take their dot product: $$\begin{aligned} \langle u_1, x_1 \rangle = u_{11}x_{11} + u_{12}x_{12} + u_{13}x_{13} + u_{14}x_{14} + u_{15}x_{15} \end{aligned}$$

Then, we see that the contribution of the first feature is $u_{11}$, because it is multiplied by that. Contribution of the second feature is $u_{12}$, because it is multiplied by that and so on. So, the signs and magnitudes of these coefficients signify the relationships between features. If they have the same sign, they try to project the data point to the same side of the principal component; and if they have the opposite side they kind of work against each other. And relative magnitudes of these components give us an idea of their similarity.

Case Study: Air Pollution in Mersin, Turkey¶

Let’s see if we can actually use PCA to detect which air quality stations are similar. We have the daily means of 5 different stations in the same city, for about a year.

From this description, we have a rough idea of the stations. But let’s use PCA to actually analyze the data:

Okay, here we see that these grouped stations are somewhat similar, as 69\%$ of the variance can be explained by one principal component.

Returning back to our loadings, and actually understanding the relationships between features; our first principal component is

$$\begin{aligned} \begin{pmatrix} 0.62 & 0.16 & 0.66 & 0.25 & -0.01 & 0.12 & 0.26 \end{pmatrix} \end{aligned}$$

From here, we observe the following groupings:

1. Stations 1, 3 ($\sim 0.6 – 0.7$)
2. Stations 4, 7 ($\sim 0.2$)
3. Stations 2, 6 ($\sim 0.1$)
4. Station 5 ($\sim 0$)

We can also see that 5 is very different than the other stations, as it has a negative sign. Let’s refresh our memories by checking the description:

We see that station 5 is the cleanest of the bunch, and is usually behaving different than them. Stations 1 and 3 are very similar, as we expected from them and so on.

We could actually decide those similarities by eye, but now we have a formal point to stand on. It is also nice that we don’t have to do it by eye as we could be talking about hundreds of stations too.

References¶

[1] Bishop, Christopher M., and Nasser M. Nasrabadi. Pattern recognition and machine learning. Vol. 4. No. 4. New York: springer, 2006.

[2] “Hava Kalitesi – İstasyon Veri İndirme | T.C. Çevre, Şehircilik ve İklim Değişikliği Bakanlığı.” 16 Apr. 2023