Replay Buffer Implementation Details
You'll learn: How to design and implement efficient replay buffers that store and sample experiences for training deep RL agents.
The Core Data Structure
A replay buffer is essentially a circular queue (also called a ring buffer) that stores experience tuples: (state, action, reward, next_state, done). When the buffer reaches capacity, new experiences overwrite the oldest ones.
Think of it like a rolling camera DVR: it constantly records, keeping only the most recent footage up to its storage limit.
Key Implementation Considerations
Storage Format
- Arrays vs Lists: Use NumPy arrays for fixed-size buffers (much faster indexing and sampling)
- State compression: Store frames as
uint8(0-255) rather thanfloat32to save 4x memory - Separate arrays: Keep one array per tuple component (states, actions, rewards, etc.) rather than storing tuples directly
Sampling Strategy
The standard approach is uniform random sampling: each stored experience has equal probability of being selected. This breaks temporal correlations in the data, which stabilizes training.
# Conceptual sampling
indices = np.random.randint(0, buffer_size, batch_size)
batch = (states[indices], actions[indices], ...)
Memory Management
- Pre-allocate full buffer capacity upfront (no dynamic resizing during training)
- Track current size and insertion position with counters
- Handle terminal states carefully: don't bootstrap from
next_stateifdone=True
Performance Tricks
- Use vectorized NumPy operations for batch assembly
- Consider GPU tensors for very large buffers (PyTorch/JAX)
- Profile memory usage: a 1M capacity buffer with 84×84×4 frames consumes ~28 GB uncompressed!
Key Takeaway: Efficient replay buffers use pre-allocated NumPy arrays in a circular queue pattern, enabling fast uniform random sampling while managing memory carefully through data compression and vectorized operations.