Linear Independence and Span

What you'll learn: How to determine if vectors are independent and understand the space they can create through combinations.

The Core Idea

Imagine you're navigating a city. If you can only move north-south OR east-west (one direction is redundant because it's just a multiple of the other), you're limited. But with both independent directions, you can reach any point in the city.

Linear independence means vectors provide genuinely different directions—none can be created by combining the others using scalar multiplication and addition. If vectors are independent, they give you maximum flexibility.

Span is the collection of all possible destinations you can reach using linear combinations (addition and scalar multiplication) of your vectors.

Understanding Linear Independence

Vectors v₁, v₂, ..., vₙ are linearly independent if the only solution to:

c₁v₁ + c₂v₂ + ... + cₙvₙ = 0

is when all scalars c₁, c₂, ..., cₙ equal zero.

If you can make the zero vector using non-zero scalars, the vectors are dependent—one is redundant.

Example: Vectors [1, 0] and [2, 0] are dependent because 2×[1, 0] - 1×[2, 0] = [0, 0]. The second vector adds no new direction.

Span: The Space Vectors Create

The span of vectors is every possible vector you can create through their linear combinations.

• Two independent vectors in 2D span the entire plane
• Three independent vectors in 3D span all of 3D space
• Two dependent vectors (pointing the same direction) only span a line

Why This Matters in ML

In machine learning, features are vectors. Independent features provide unique information. Dependent features are redundant—they waste computational resources without improving your model. Understanding span helps you know what your feature space can represent.

Key Takeaway: Linear independence means vectors provide unique directions; span is all the points those vectors can reach through combinations—both concepts help identify redundant vs. valuable features in ML.