Mathematics
Principal Component Analysis for Dimension Reduction
Quick fact
PCA was invented in 1901 by Karl Pearson, but it only became widely used after computers made the calculations practical.
Why this is interesting
Imagine you have a dataset with hundreds of measurements for each item, but you suspect only a few underlying trends matter. How can you simplify it without losing the essence?
Read the full explanation
Understanding Principal Component Analysis for Dimension Reduction
Think of a cloud of points in space. PCA finds the direction in which the points spread out the most. That direction is the first principal component. Then it finds a second direction, perpendicular to the first, that captures the most of the remaining spread, and so on. These directions are like new axes. By taking only the first few axes, you can project your data into a lower-dimensional space that still shows the main structure. This is like flattening a 3D object into a 2D shadow from the angle that shows the most detail.
A deeper explanation
PCA works by computing the covariance matrix of the data, which describes how variables vary together. The eigenvectors of this matrix point in the directions of maximum variance, and the eigenvalues tell how much variance each direction captures. The eigenvector with the largest eigenvalue is the first principal component. By projecting the data onto the top k eigenvectors, you get a lower-dimensional representation that retains the maximum possible variance. This is equivalent to finding the best-fitting linear subspace in terms of least-squared error. PCA is widely used for data compression, noise reduction, and visualization of high-dimensional data.