Course contentsShow
← Machine Learning and Deep Learning
Lesson 2245 of 3,53848. Deep Reinforcement Learning: Value-BasedFree lesson

Training Loop Structure

Organize the complete training loop: environment steps, experience collection, batch sampling, and network updates.

Training Loop Structure

What you'll learn: How to organize a complete Deep Q-Network training loop that orchestrates environment interaction, experience collection, sampling, and neural network optimization.

The Complete Picture

You've learned all the individual components of DQN—now it's time to wire them together into a cohesive training loop. Think of this like conducting an orchestra: each instrument (replay buffer, target network, epsilon-greedy selection) plays its part, but you need a conductor to coordinate when and how they perform together.

The Core Training Loop

A well-structured DQN training loop follows this rhythm:

  1. Environment Steps: Agent observes state, selects action using epsilon-greedy policy, executes action, receives reward and next state
  2. Experience Collection: Store the transition (state, action, reward, next_state, done) in the replay buffer
  3. Batch Sampling: Once enough experiences exist, sample a random minibatch from the replay buffer
  4. Network Update: Compute target Q-values using the target network, calculate TD-error loss, backpropagate gradients, update the main Q-network
  5. Target Network Sync: Periodically copy weights from the main network to the target network
  6. Logging & Evaluation: Track episode rewards, loss values, and epsilon decay

Key Design Considerations

When to update? Don't update on every step—wait until the replay buffer has sufficient data, then update every few steps or once per episode.

Outer vs inner loops: The outer loop iterates over episodes (full environment runs), while the inner loop handles individual timesteps within each episode.

Decoupling: Separate environment interaction from learning. You might collect 4 steps, then perform 1 gradient update—this ratio is a hyperparameter.

Key Takeaway: The training loop structure orchestrates all DQN components into a synchronized flow: interact with the environment, store experiences, sample batches, compute losses, update networks, and repeat until convergence.