Course contentsShow
← Machine Learning and Deep Learning
Lesson 371 of 3,5388. Unsupervised Learning: ClusteringFree lesson

Covariance Structure Constraints

Full, diagonal, spherical, and tied covariance matrices and their impact on cluster shapes and efficiency.

Covariance Structure Constraints

What you'll learn: How different covariance matrix types in Gaussian Mixture Models control cluster shapes and computational efficiency.

The Core Idea

When fitting Gaussian Mixture Models, each Gaussian component has a covariance matrix that determines its shape and orientation. However, estimating a full covariance matrix for high-dimensional data requires many parameters and lots of computation. By imposing constraints on the covariance structure, we trade modeling flexibility for efficiency and numerical stability.

The Four Common Constraints

Full Covariance

Each cluster can be any ellipsoid with any orientation. Most flexible but computationally expensive—estimates all possible correlations between features.

Diagonal Covariance

Clusters are axis-aligned ellipsoids. Assumes features are independent within each cluster. Faster to compute and uses fewer parameters.

Spherical Covariance

Clusters are perfect spheres (same variance in all directions). Most restrictive but extremely efficient—just one variance parameter per cluster.

Tied Covariance

All clusters share the same covariance structure but can be centered at different locations. Reduces parameters while allowing orientation.

Practical Impact

Think of it like buying furniture: full covariance is custom-made (expensive, perfect fit), diagonal is adjustable (width/height independent), spherical is one-size (cheap, fast), and tied means all rooms get identical furniture (consistent but less flexible).

Choosing the right constraint depends on your data's true structure and computational budget. If clusters are truly circular, spherical constraints prevent overfitting. If they're stretched and rotated differently, you'll need full covariance.

Key Takeaway: Covariance structure constraints balance model expressiveness against computational cost—full allows arbitrary ellipsoids, diagonal restricts to axis-aligned, spherical forces circles, and tied shares structure across all components.