ModelRefs / Principal Component Analysis — Tutorial
Reduce dimensions, remove noise, and visualise high-dimensional data — the workhorse of unsupervised feature extraction
Principal Component Analysis — Tutorial: This tutorial provides a structured implementation path with prerequisites, steps, checkpoints, and related references. Read the complete sequence before applying commands or configuration in production.
Principal Component Analysis — Tutorial: Adapt examples to the versions, security boundaries, data policy, and failure-handling requirements of your system. Validate intermediate outputs and keep a rollback path for changes that affect users or stored data.
Principal Component Analysis — Tutorial: Tutorial examples demonstrate a technique; they do not prove reliability, compliance, performance, or suitability for a workload. Use current primary documentation and test the final system under representative conditions.
PCA finds the directions (principal components) of greatest variance in your data and projects data onto them. The first principal component captures the most variance, the second captures the most remaining variance while being orthogonal to the first, and so on.
Why reduce dimensions? Three reasons: speed (fewer features → faster training), noise removal (low-variance directions often contain noise), and visualisation (project to 2D to see cluster structure humans can interpret).
PCA is a linear transformation. It does NOT discard features — it combines all original features into new ones. A component might be "0.8 × age − 0.3 × income + 0.5 × tenure" — a weighted mix. The weights are called loadings.
Step 1: Centre the data — subtract the column mean so each feature has mean 0.
Step 2: Compute the covariance matrix Σ = XᵀX / (n−1). This captures how features vary together.
Step 3: Eigen-decompose Σ. The eigenvectors are the principal component directions; the eigenvalues are the variances along those directions.
Step 4: Sort by eigenvalue descending. The first eigenvector (highest eigenvalue) is PC1.
Step 5: Project data onto the top K eigenvectors: X_reduced = X @ V[:, :K].
Explained variance ratio: eigenvalue_k / Σ eigenvalues. If the first 2 PCs explain 95% of total variance, you can safely drop the rest.
The steps above describe PCA via eigendecomposition of the covariance matrix — how the method is usually taught. But scikit-learn's PCA computes components via singular value decomposition (SVD) of the centred data matrix directly: X = U · Σ · Vᵀ. The right singular vectors (V) are the same principal component directions you'd get from eigendecomposition — the math is equivalent — but SVD skips forming the covariance matrix, which is more numerically stable and cheaper when you have many features.
This is a real implementation decision, not just an optimisation detail: it's why PCA(svd_solver=...) exists (see the Syntax Reference below), and why PCA "just works" on data with many features without you ever forming an n_features × n_features matrix by hand.
Four points, one feature pair, so you can check every step before trusting the code — A(2,4), B(4,6), C(6,8), D(8,10):
1. Centre: mean = (5, 7). Centred points: (−3,−3), (−1,−1), (1,1), (3,3). 2. Covariance: Σ = [[6.67, 6.67], [6.67, 6.67]] — x₁ and x₂ move together perfectly (correlation = 1), which is why this toy example has only one real direction of variance. 3. Eigen-decompose: eigenvalues ≈ 13.33 and 0. The eigenvector for 13.33 is (0.707, 0.707) — the 45° line. 4. PC1 = (0.707, 0.707). Explained variance ratio = 13.33 / (13.33 + 0) = 100%. 5. Project: each point's PC1 score is its distance along that 45° line — A ≈ −4.24, B ≈ −1.41, C ≈ 1.41, D ≈ 4.24.
The takeaway before you touch real data: PCA didn't lose anything here — one number per point (the PC1 score) fully reconstructs both original features, because x₂ was always exactly x₁ + 2.
PCA helps when: features are highly correlated (multicollinearity), you need to visualise clusters in high-dimensional data, or you want to compress features before a distance-based model (KNN, SVM).
PCA hurts when: the relationship between features and target is non-linear (PCA is linear — reach for non-linear methods like UMAP or t-SNE for visualisation instead), you need interpretable features (PCA components are weighted mixtures, not original features), or the target is encoded in low-variance directions (which PCA discards by design).
When you have more features than samples (p ≫ n) — common in genomics, text embeddings, or wide sensor data — forming the n_features × n_features covariance matrix from Step 2 is often computationally infeasible even though SVD keeps the component computation itself tractable. In that regime, IncrementalPCA processes data in batches without holding the full matrix in memory, and KernelPCA handles non-linear structure the linear method can't. Both are one-line swaps for PCA in the syntax below, not separate algorithms to relearn.
Rule of thumb: try PCA if you have >20 correlated features and your model training is slow. Measure accuracy before and after — if accuracy drops more than 2%, use fewer components or skip PCA. The data-analysis-assistant workflow has a runnable version of this before/after check.
| Function | Signature | Description |
|---|---|---|
PCA | from sklearn.decomposition import PCA; pca = PCA(n_components=2) | Principal Component Analysis. n_components: number of PCs to keep (int or float 0–1 for variance ratio) |
svd_solver | PCA(n_components=k, svd_solver="auto") | "full" (exact, best for small/dense data), "randomized" (approximate, much faster for large n_features), "auto" (sklearn picks based on data shape). Encodes the eigendecomposition-vs-SVD tradeoff described above |
pca.explained_variance_ratio_ | pca.explained_variance_ratio_ # array of shape (n_components,) | Fraction of total variance explained by each component |
pca.components_ | pca.components_ # shape (n_components, n_features) | Principal component loadings — how much each original feature contributes to each PC |
pca.inverse_transform | pca.inverse_transform(X_reduced) | Reconstructs approximate original features from reduced data — useful for measuring information loss directly, not just via explained variance |
PCA for visualisation and compression
from sklearn.decomposition import PCA
from sklearn.datasets import load_digits
from sklearn.preprocessing import StandardScaler
import numpy as np
digits = load_digits()
X, y = digits.data, digits.target # shape: (1797, 64)
# Always scale before PCA
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)
# How many components for 95% variance?
pca_full = PCA().fit(X_scaled)
cumvar = np.cumsum(pca_full.explained_variance_ratio_)
n_95 = np.argmax(cumvar >= 0.95) + 1
print(f"Components for 95% variance: {n_95} (out of 64)")
# 2D projection for visualisation
pca2 = PCA(n_components=2)
X_2d = pca2.fit_transform(X_scaled)
print(f"Explained variance (2 PCs): {pca2.explained_variance_ratio_.sum():.2%}")
print(f"PC1 top feature indices: {np.argsort(np.abs(pca2.components_[0]))[-3:]}")
# Compression + reconstruction
pca_k = PCA(n_components=n_95)
X_compressed = pca_k.fit_transform(X_scaled)
X_reconstructed = pca_k.inverse_transform(X_compressed)
mse = np.mean((X_scaled - X_reconstructed) ** 2)
print(f"Reconstruction MSE ({n_95} components): {mse:.4f}")cumvarChallenge: Build an sklearn Pipeline that applies PCA before a logistic regression classifier and tune n_components.
from sklearn.decomposition import PCA
from sklearn.linear_model import LogisticRegression
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
from sklearn.model_selection import GridSearchCV
from sklearn.datasets import load_breast_cancer
X, y = load_breast_cancer(return_X_y=True)
pipe = Pipeline([
("scaler", StandardScaler()),
("pca", PCA()),
("clf", LogisticRegression(max_iter=1000)),
])
param_grid = {"pca__n_components": [2, 5, 10, 20, 30]}
grid = GridSearchCV(pipe, param_grid, cv=5, scoring="accuracy")
grid.fit(X, y)
print(f"Best n_components: {grid.best_params_}")
print(f"Best CV accuracy: {grid.best_score_:.4f}")1. Explained variance ratio of [0.45, 0.28, 0.12] means:
0.45 + 0.28 + 0.12 = 0.85. Each number is that PC's share of total variance — not a count of points, an accuracy figure, or a feature count. The remaining 15% is in the discarded components.
2. PCA requires feature scaling because:
PCA maximises variance, so a feature measured in the thousands (income) will swamp one measured in single digits (age) unless both are standardised first. Scaling equalises units — it does not filter noise (that's what dropping low-variance components does, afterward).
3. The first principal component is the direction that:
Capturing the most variance is PC1's defining property. Minimising squared perpendicular distance to the line is the equivalent framing; minimising distance to points (option B) describes regression, and separating classes (option D) is supervised LDA — PCA never looks at labels.
Related concepts named above that do not yet have their own tutorial page: Covariance and correlation, Eigenvectors and eigenvalues, t-SNE, UMAP, Kernel PCA, Incremental PCA, Linear Discriminant Analysis (LDA), Feature importance vs. loadings.
Related workflow: data-analysis-assistant.
Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Principal Component Analysis — Tutorial.