The Coda of Principal Component Analysis

Principal component analysis (PCA) is a statistical method that is meant to simplify large data sets by reducing the number of variables. There have been several mathematical attempts to maximize the variance accounted for in a data set. From these calculations, variance–covariance matrices and correlation structures can emerge.

This Q&A overview of PCA serves as a technical guide that explains the complex inner workings of the PCA by working through rigorous equations.

Why derive principal component analysis (PCA) using elementary algebra instead of standard matrix notation?

Most literature presents PCA strictly using matrix algebra because writing out full multivariate equations in standard algebraic notation is unwieldy.¹ However, matrix operations can act as a conceptual barrier, hiding what actually occurs "under the hood.”¹ By stepping back to fundamental principles of elementary algebra and basic calculus, the underlying mathematical behavior becomes intuitively transparent.¹ Matrix equations can then serve as useful landmark comparisons alongside the algebraic steps.¹

How does PCA differ fundamentally from other approximation techniques like Fourier or Taylor series?

Other approximation techniques such as the Fourier series (trigonometric sines and cosines) and Taylor series (polynomials) reconstruct data using predefined, a priori mathematical functions.¹ On the other hand, PCA does not do that. Instead, PCA generates empirical functions derived directly from the underlying data matrix.¹ Another major difference is how these techniques construct their algorithms. Principal components are least-squares and maximum-variance estimators; trigonometric (Fourier) functions are also mutually orthogonal, that property does not distinguish the two. The real distinction is that Fourier and Taylor functions are fixed, predetermined functions chosen independent of the data, whereas principal components are empirical functions derived from and optimized to the specific data set, which is why they alone carry the maximum-variance, least-squares property.¹ Principal components, as a result, reconstruct the specific data set with smaller sum-squared error than any other set of functions.¹

Why is mean-centering data an essential prerequisite step before deriving principal components?

Subtracting the mean spectrum from every sample spectrum sets the mean of the entire data set to zero.^1,2 Because the data set mean is constant across all samples, its variance is zero. Therefore, this process removes a useless term without altering the remaining variance structure.^1,2 Notation is also simplified. This means that mean-centering allows for simple sums of squares and cross-products to directly represent variances and covariances.^1–3

What "trivial solutions" occur in unconstrained algebraic derivations of PCA, and how are they avoided?

There are three unhelpful trivial solutions that occur in unconstrained algebraic derivations of PCA. These three solutions are the zero variance solution; the single-variable solution; and the infinite variance solution.

Let’s explore these in depth. For the zero variance solution, all loading coefficients equal zero, giving zero variance.^1–3 For the single-variable solution, setting all loadings to zero except one variable, creating an artificial zero-error fit for that single variable while failing to fit the data set.² For the infinite variance solution, loading coefficients grow infinitely large without bound.^1,3,4 To prevent these collapses, a constraint is introduced using a Lagrange multiplier from the calculus of variations, enforcing that the sum of squared loading coefficients equals unity.⁴

How does the correlation structure of multivariate data naturally emerge during the mathematical derivation?

Univariate attempts treat variables independently, failing to capture correlations across variables.³ When formulating the multivariate sum-squared error across all variables simultaneously and expanding the squared terms, cross-product sum terms appear.³ Differentiating the constrained sum of squares accounted for with respect to the loading coefficients naturally constructs the variance-covariance matrix.³ This demonstrates mathematically that PCA inherently subsumes the correlation structure of the data.^3–5

How are the actual loading coefficients solved for once the variance-covariance matrix appears?

The article moves directly from “a variance-covariance matrix emerges” to “how subsequent components are calculated” without explaining how the first component’s loadings are actually found. The Lagrange-multiplier setup produces a system of homogeneous equations that has a non-trivial solution only when the determinant of that system equals zero; expanding that determinant yields the characteristic (eigenvalue) equation, whose roots are the eigenvalues of the variance-covariance matrix.⁵ Each eigenvalue is substituted back in to solve for its corresponding eigenvector, the loading coefficients, with the largest eigenvalue giving the first principal component.⁵ This is the step that connects the constrained-optimization setup to an actual, computable answer.

How are subsequent principal components calculated after the first component is found?

Mathematically, additional components can be derived by adding a second Lagrange multiplier constraint enforcing orthogonality to the first loading vector.⁶ In practice, the contribution of the first principal component (the score multiplied by the normalized loading vector) is subtracted row-by-row and wavelength-by-wavelength from the data matrix.⁶ The full PCA algorithm is then re-run on the resulting residual error matrix to calculate the second principal component, repeating the process iteratively for remaining components.⁶

References

1. Workman, Jr., J. The Long, Complicated, Tedious, and Difficult Route to Principal Components: Part I. Spectroscopy 2007, 22 (9). Available at: https://www.spectroscopyonline.com/view/long-complicated-tedious-and-difficult-route-principal-components-part-i
2. Workman, Jr., J.; Mark, H. The Long, Complicated, Tedious, and Difficult Route to Principal Components: Part II. Spectroscopy 2008, 23 (2). Available at: https://www.spectroscopyonline.com/view/long-complicated-tedious-and-difficult-route-principal-components-part-ii
3. Workman, Jr., J.; Mark, H. The Long, Complicated, Tedious, and Difficult Route to Principal Components: Part III. Spectroscopy 2008, 23 (5). Available at: https://www.spectroscopyonline.com/view/long-complicated-tedious-and-difficult-route-principal-components-part-iii
4. Workman, Jr., J.; Mark, H. The Long, Complicated, Tedious, and Difficult Route to Principal Components: Part IV. Spectroscopy 2008, 23 (6). Available at: https://www.spectroscopyonline.com/view/long-complicated-tedious-and-difficult-route-principal-components-part-iv
5. Workman, Jr., J.; Mark, H. The Long, Complicated, Tedious, and Difficult Route to Principal Components: Part V. Spectroscopy 2009, 24 (1). Available at: https://www.spectroscopyonline.com/view/long-complicated-tedious-and-difficult-route-principal-components-part-v
6. Workman, Jr., J.; Mark, H. The Long, Complicated, Tedious, and Difficult Route to Principal Components: Part VI. Spectroscopy 2009, 24 (2). Available at: https://www.spectroscopyonline.com/view/long-complicated-tedious-and-difficult-route-principal-components-part-vi