Linear Algebra — The Language of Dimensions

🔬 Scientist Pure mathematics — no exam syllabus

Linear Algebra — The Language of Dimensions

Linear algebra is the mathematics of the 21st century. From Google's PageRank to Netflix recommendations, from quantum mechanics to facial recognition — it's all linear algebra underneath. It's the study of vectors, matrices, and the transformations between them. And at its heart lies a simple question: what happens to space when you stretch, rotate, or project it?

"Linear algebra is the one subject that every scientist, engineer, and data scientist needs to know. It is the language of modern computation."

— Gilbert Strang, MIT

Part 1

➡️ Vectors — Arrows in Space

The Geometry of Numbers

Why (3, 4) Is an Arrow, Not Just a Point

A vector is an arrow with a direction and a length. The vector (3, 4) means "go 3 units right and 4 units up." Vectors can be added: (3, 4) + (1, 2) = (4, 6). And scaled: 2 × (3, 4) = (6, 8). This is the foundation — vectors combine like forces, like velocities, like everything in the physical world. A vector space is simply the collection of all vectors you can reach by adding and scaling a given set of "building block" vectors.

Vectors are the atoms of linear algebra. Everything else is built from them.

The Dot Product — Measuring Alignment

The dot product v · w = v₁w₁ + v₂w₂ + ... + vₙwₙ tells you how much two vectors point in the same direction. If v · w = 0, they're perpendicular (orthogonal). If v · w = |v||w|, they're perfectly aligned.

Dot Product = Projection

v · w = |v| |w| cos θ

The dot product is the product of lengths times the cosine of the angle between them.

Example: Vectors in 3D

v = (3, 4, 0), w = (0, 0, 5).
v · w = 3×0 + 4×0 + 0×5 = 0. They're perpendicular!
|v| = √(3²+4²) = 5, |w| = 5. So cos θ = 0, θ = 90°.

Span, Basis, and Dimension

The span of a set of vectors is all vectors you can reach by linear combinations (adding and scaling). A basis is a minimal set that spans the space — no redundancies. The number of vectors in a basis is the dimension.

✨

Intuition: Why 3D Needs 3 Basis Vectors ▶

In 3D space, you need three independent directions: forward/back, left/right, up/down. No combination of just "forward" and "right" can give you "up." That's what linear independence means — each basis vector adds a new direction that can't be reached using the others. The standard basis in 3D: î = (1,0,0), ĵ = (0,1,0), k̂ = (0,0,1). Every vector in 3D can be written uniquely as a combination of these three.

Part 2

🧮 Matrices — The Algebra of Transformations

Machines That Move Space

A Matrix Is a Function That Eats Vectors

A matrix is a rectangular array of numbers, but don't be fooled by its simplicity. Think of it as a machine: you feed in a vector, and it spits out another vector. The matrix rotates space, or stretches it, or squishes it onto a line. The product Av = b means "apply transformation A to vector v, get vector b." Matrix multiplication (AB) means "apply B first, then A" — composition of transformations. This is the deep idea: matrices are linear functions.

A matrix is not just a table of numbers — it's a description of how to transform space.

Matrix Multiplication as Composition

If A rotates 90° and B stretches horizontally, then AB first stretches then rotates (right to left). The order matters — AB ≠ BA in general.

2×2 Rotation by θ

[cosθ, -sinθ; sinθ, cosθ]

Identity

I = [1 0; 0 1]

Horizontal Stretch by k

[k 0; 0 1]

Shear

[1 k; 0 1]

The Inverse — Undoing a Transformation

If A doubles the x-direction, A⁻¹ halves it. If A rotates by 30°, A⁻¹ rotates by -30°. The product A × A⁻¹ = I, the identity (which does nothing). A matrix has an inverse only if it doesn't collapse space — a 2D-to-2D transformation that squishes everything onto a line loses information and is not invertible.

The Determinant — How Much Does Space Stretch?

The determinant det(A) tells you the factor by which A changes area (in 2D) or volume (in 3D). If det(A) = 2, areas double. If det(A) = 0, space is collapsed (not invertible). If det(A) is negative, orientation is flipped (like a mirror).

Determinant of 2×2

det(a b; c d) = ad − bc

The signed area of the parallelogram formed by the columns.

Systems of Equations — The Original Problem

Linear algebra was born to solve systems like: 2x + 3y = 7, 4x − y = 1. In matrix form: A x = b, where A = [2 3; 4 -1], x = (x, y), b = (7, 1). The solution is x = A⁻¹b, found via Gaussian elimination — systematically eliminating variables.

✨

Gaussian Elimination — The Workhorse Algorithm ▶

Solve: 2x + 3y = 7, 4x − y = 1.

Step 1: Write the augmented matrix:
[2 3 | 7; 4 -1 | 1]

Step 2: Eliminate x from row 2: R₂ ← R₂ − 2R₁ (because 2×2 = 4).
[2 3 | 7; 0 -7 | -13]

Step 3: Back-substitute: -7y = -13 → y = 13/7.
2x + 3(13/7) = 7 → 2x = 7 − 39/7 = 10/7 → x = 5/7.

Solution: (x, y) = (5/7, 13/7).
This algorithm scales to thousands of equations — it's how computers solve linear systems.

Part 3

🔮 Eigenvalues — The Hidden Directions

The Directions That Don't Bend

Eigenvectors: The Special Vectors That Only Stretch

When a matrix A acts on a vector v, the result is usually a completely different direction. But for some special vectors — eigenvectors — the result is just a scaled version of the same vector: Av = λv. The scalar λ is the eigenvalue. These are the "natural directions" of the transformation. In a rotating planet, the axis of rotation is an eigenvector (it doesn't change direction). In a vibrating bridge, the normal modes are eigenvectors. In Google's PageRank, the ranking vector is the dominant eigenvector of the web's link matrix.

Eigenvectors are the directions that a transformation treats most simply — they reveal the hidden structure of a matrix.

Finding Eigenvalues

From Av = λv, we get (A − λI)v = 0. For non-zero v to exist, the matrix (A − λI) must be singular — its determinant must be zero. This gives the characteristic polynomial.

Characteristic Equation

det(A − λI) = 0

The roots λ are the eigenvalues. For an n×n matrix, this is an nth-degree polynomial.

Example: Find Eigenvalues of A = [2 1; 1 2]

det(A − λI) = det(2-λ, 1; 1, 2-λ) = (2-λ)² − 1 = λ² − 4λ + 3 = 0.
λ = 1 or λ = 3.

For λ = 3: (A − 3I)v = 0 → [-1 1; 1 -1]v = 0 → v = (1, 1).
For λ = 1: (A − I)v = 0 → [1 1; 1 1]v = 0 → v = (1, -1).

Check: A(1,1) = (3,3) = 3×(1,1) ✓. A(1,-1) = (1,-1) = 1×(1,-1) ✓.

Diagonalization — The Ultimate Simplification

If a matrix has enough eigenvectors, you can write A = PDP⁻¹, where D is a diagonal matrix of eigenvalues and P contains the eigenvectors. This is like changing to a coordinate system where the transformation is just scaling along each axis — the simplest possible description.

✨

Why Diagonalization Matters ▶

Computing A¹⁰⁰ directly is expensive. But if A = PDP⁻¹, then:
A¹⁰⁰ = (PDP⁻¹)¹⁰⁰ = PD¹⁰⁰P⁻¹
D¹⁰⁰ just raises each diagonal eigenvalue to the 100th power — trivial!

This powers entire fields: Markov chains, population dynamics, quantum mechanics, and differential equations all rely on this trick.

The Spectral Theorem — Symmetric Matrices Are Special

For symmetric matrices (A = Aᵀ), eigenvalues are always real, and eigenvectors are orthogonal. This is why covariance matrices, correlation matrices, and distance matrices — all symmetric — have such nice properties. The Spectral Theorem says every symmetric matrix can be written as A = QΛQᵀ where Q is an orthogonal matrix (its columns are orthonormal eigenvectors).

Singular Value Decomposition — The Master Factorization

The SVD factorizes any matrix (not just square) into A = UΣVᵀ. U contains left singular vectors, V contains right singular vectors, and Σ contains singular values (like eigenvalues but for non-square matrices). The SVD is the crown jewel of linear algebra — it works for every matrix, reveals its rank, and gives the best low-rank approximation.

SVD: Any Matrix, Any Shape

A = U Σ Vᵀ

U and V are orthogonal, Σ is diagonal. The singular values σ₁ ≥ σ₂ ≥ ... ≥ σᵣ > 0 reveal the "importance" of each dimension.

Part 4

🌍 Linear Algebra in the Real World

The Unreasonable Effectiveness

Why the 19th Century's Nicest Math Powers 21st-Century Tech

Linear algebra was developed largely in the 1800s — before computers, before data, before the internet. Yet it turned out to be the perfect language for the digital age. Why? Because digital data is naturally vector-shaped. A grayscale image is a matrix of pixel values. A database of users is a matrix of features. A website's links form a matrix. And the operations we want — find patterns, reduce noise, rank importance — are all linear algebra.

Linear algebra is not just math — it's the operating system of modern computation.

PCA — Dimensionality Reduction

Suppose you have 1000-dimensional data (1000 features per sample). Much of it is noise. Principal Component Analysis (PCA) finds the directions of highest variance — the eigenvectors of the covariance matrix — and projects the data onto the top few. This is the SVD in action: the principal components are the right singular vectors, and the singular values tell you how much variance each component captures.

PCA in Practice

A face recognition dataset has 10,000 pixels per image. PCA reduces each image to just 50-100 "eigenface" coefficients — compressing by 99% while retaining 95% of the information. This is how early facial recognition worked.

Google PageRank — The $100B Eigenvector

PageRank models the web as a matrix: entry Aᵢⱼ = 1 if page j links to page i, weighted by the out-degree of j. The ranking vector r is the dominant eigenvector: r is the solution to r = Ar, or equivalently r is the eigenvector with eigenvalue 1. This is also the stationary distribution of a random surfer — someone clicking random links eventually spends proportion rⱼ of their time on page j. That proportion is the PageRank. This single linear algebra idea built Google.

"PageRank can be computed by finding the principal eigenvector of the link matrix."

— Brin & Page, 1998

Least Squares — Fitting Lines and Curves

When you have more equations than unknowns (overdetermined system), there's usually no exact solution. The least squares method finds the x that minimizes ||Ax − b||². The solution is x = (AᵀA)⁻¹Aᵀb — the normal equations. This is the math behind every regression line, every curve fit, every machine learning model that minimizes squared error.

✨

The Geometry of Least Squares ▶

Geometrically, Ax = b has a solution only if b is in the column space of A. If not, the best we can do is project b onto the column space. The projection is A x̂ where x̂ = (AᵀA)⁻¹Aᵀb, and the residual b − A x̂ is orthogonal to the column space.

This is why it's called "least squares" — we're minimizing the squared length of the error vector, which is the sum of squared residuals. Linear regression, polynomial fitting, and even neural network training all use this principle.

Image Compression via SVD

A 1000×1000 grayscale image is a matrix of 1 million numbers. The SVD gives A = σ₁u₁v₁ᵀ + σ₂u₂v₂ᵀ + ... + σᵣuᵣvᵣᵀ. The early terms (large σ) capture the broad structure; later terms capture fine detail. By keeping only the top k singular values, you get the best rank-k approximation to the image. A 1000×1000 image compressed to k=50 uses only 50×(1000+1000+1) ≈ 100,000 numbers — a 10× compression with barely visible quality loss.

Quantum Mechanics — The Ultimate Linear Theory

In quantum mechanics, the state of a particle is a vector in an infinite-dimensional Hilbert space. Observables (position, momentum, energy) are linear operators (matrices). The possible measurement values are eigenvalues. The state evolution is given by Schrödinger's equation, which is a linear differential equation. Everything in quantum mechanics is linear algebra — it's the language of reality at its most fundamental level.

Quantum Mechanics = Linear Algebra

H |ψ⟩ = E |ψ⟩

Schrödinger's equation is an eigenvalue problem. The Hamiltonian H has eigenvalues E — the possible energy levels. The eigenvectors |ψ⟩ are the stationary states.

🎯 Practice — Check Your Linear Intuition

These puzzles build your linear algebra intuition — no calculator needed.

← Back to Math Map