t-SNE: The Cost Function and Optimization
What You'll Learn
You'll understand how t-SNE measures the quality of its embedding and optimizes it using Kullback-Leibler divergence and gradient descent.
The Core Idea
In the previous lesson, you learned that t-SNE creates probability distributions over pairs of points in both high-dimensional and low-dimensional spaces. Now we need a way to measure how well the low-dimensional embedding preserves the neighborhood structure from the original data.
Kullback-Leibler Divergence as the Cost Function
t-SNE uses Kullback-Leibler (KL) divergence to measure how different two probability distributions are. Think of it like measuring the "disappointment" when you use one distribution (the low-dimensional embedding) to approximate another (the high-dimensional similarities).
The KL divergence for t-SNE is:
KL(P || Q) = Σᵢ Σⱼ pᵢⱼ log(pᵢⱼ / qᵢⱼ)
Where:
pᵢⱼ= similarity probability in high-dimensional spaceqᵢⱼ= similarity probability in low-dimensional space
When the two distributions match perfectly, KL divergence equals zero. The more they differ, the larger the value.
Gradient Descent Optimization
t-SNE minimizes this KL divergence using gradient descent. It iteratively adjusts the positions of points in the low-dimensional space, moving them to reduce the "cost" (the mismatch between distributions). The gradient tells each point which direction to move to better preserve its neighborhood relationships.
The process is computationally intensive because it must consider all pairwise relationships, which is why t-SNE can be slow on large datasets.
Key Takeaway: t-SNE uses KL divergence to measure how well the low-dimensional embedding preserves high-dimensional similarities, then uses gradient descent to minimize this divergence by iteratively adjusting point positions.