Course contentsShow
← Machine Learning and Deep Learning
Lesson 1015 of 3,53823. Recurrent Neural NetworksFree lesson

LSTM Forget Gate

How the forget gate decides which information to remove from the cell state using a sigmoid-gated element-wise multiplication.

LSTM Forget Gate

What You'll Learn

You'll learn how the forget gate selectively erases outdated information from the LSTM cell state using a sigmoid-gated multiplication operation.

The Problem of Irrelevant Memory

Imagine reading a story where characters change from chapter to chapter. Your brain naturally "forgets" details about old characters when new ones appear—you don't need to remember everything forever. LSTMs need the same ability: a mechanism to actively decide what to forget from the cell state.

How the Forget Gate Works

The forget gate is the first gate that processes information in each LSTM time step. Here's the step-by-step process:

  1. Input combination: The gate receives two inputs—the current input x_t and the previous hidden state h_{t-1}

  2. Sigmoid activation: These inputs pass through a sigmoid function (σ), producing values between 0 and 1:

    f_t = σ(W_f · [h_{t-1}, x_t] + b_f)
    
  3. Element-wise multiplication: The forget gate output f_t multiplies the previous cell state C_{t-1} element-by-element

  4. Selective forgetting:

    • Values close to 1 mean "keep this information"
    • Values close to 0 mean "forget this information"
    • Values in between partially retain information

An Analogy

Think of the cell state as a whiteboard with notes. The forget gate is like an eraser with variable transparency—it can completely erase parts (0), leave other parts untouched (1), or partially fade sections (values between 0 and 1).

Key Takeaway: The forget gate uses sigmoid activation to produce values between 0 and 1, which multiply the cell state element-wise, allowing the LSTM to selectively remove outdated information while preserving relevant memories.