Course contentsShow
← Machine Learning and Deep Learning
Lesson 1006 of 3,53823. Recurrent Neural NetworksFree lesson

Truncated Backpropagation Through Time

Limiting gradient flow to a fixed number of steps to make training computationally feasible.

Truncated Backpropagation Through Time

What you'll learn: How to make training recurrent neural networks computationally practical by limiting how far back gradients flow through time.

The Core Problem

You already know that Backpropagation Through Time (BPTT) unrolls an RNN across all time steps and computes gradients backward through the entire sequence. But what if your sequence has 10,000 time steps? Computing and storing gradients through all those steps becomes computationally prohibitive—you'd run out of memory and time.

The Solution: Truncation

Truncated BPTT limits gradient flow to a fixed number of recent time steps (say, 50 or 100), even when your sequence is much longer. Instead of backpropagating through the entire sequence, you only go back a fixed "window" of steps.

How It Works

Think of it like looking through a moving window on a train journey. You can only see and remember the last few stations you passed, not the entire route from the starting point. Similarly, truncated BPTT:

  1. Processes the sequence forward normally through all time steps
  2. When computing gradients, only backpropagates through the last k steps
  3. Moves the window forward and repeats

The forward pass still maintains the hidden state across the entire sequence, preserving long-term dependencies in predictions. But the backward pass is "truncated" to make training feasible.

The Trade-off

You gain computational efficiency but sacrifice some gradient information from distant past steps. This is often an acceptable trade-off—the truncated gradients still capture most important patterns, and the alternative (not training at all due to resource constraints) is worse.

Key Takeaway: Truncated Backpropagation Through Time makes RNN training practical for long sequences by limiting gradient computation to a fixed window of recent time steps, trading some gradient accuracy for massive computational savings.