Temporal Smoothing and Tracking
What you'll learn: How to eliminate jittery predictions and maintain stable object identities across video frames using temporal filtering and tracking algorithms.
The Problem: Prediction Jitter
When running frame-by-frame inference on video, you've likely noticed predictions "jumping around" — bounding boxes flicker, classifications oscillate between classes, and detected objects appear and disappear randomly. This happens because each frame is processed independently without considering temporal context.
Temporal Smoothing
Temporal smoothing applies filtering across consecutive frames to stabilize predictions. Think of it like noise-canceling headphones for your model's output — instead of believing every single prediction, you average or weight recent history.
Common approaches:
- Moving average: Average the last N predictions (positions, class scores)
- Exponential smoothing: Weight recent frames more heavily than distant ones
- Kalman filtering: Use motion models to predict where objects should be, correcting for measurement noise
For example, if a bounding box jumps 5 pixels left, 3 right, then 4 left across three frames, smoothing reveals the object is actually moving slowly leftward rather than spastically bouncing.
Object Tracking
Tracking maintains consistent identities for objects across frames. Instead of treating frame 10's "person" as unrelated to frame 9's "person," tracking algorithms link them together.
Key techniques:
- IoU matching: Link detections with high overlap across frames
- Feature-based tracking: Match objects using appearance embeddings
- Motion prediction: Anticipate where objects will move next
Tracking prevents ID switches (person #3 becoming person #7) and handles temporary occlusions by predicting positions even when detection fails.
The Combination Effect
Together, smoothing and tracking create stable, production-grade video inference: bounding boxes glide smoothly, classifications remain consistent, and objects maintain their identity throughout the video — even if your per-frame detector occasionally hiccups.
Key Takeaway: Temporal smoothing filters noisy predictions across frames while tracking maintains object identities, transforming jittery frame-by-frame inference into stable, coherent video understanding.