Course contentsShow
← Machine Learning and Deep Learning
Lesson 812 of 3,53819. PyTorch FundamentalsFree lesson

Registering Buffers for Non-Learnable State

Learn how register_buffer() stores tensors that should move with the model but aren't learnable parameters.

Registering Buffers for Non-Learnable State

What You'll Learn

You'll learn how to store tensors in your model that need to move with it (like to GPU) but shouldn't be updated during training.

The Core Problem

Sometimes your model needs to keep track of tensors that aren't learnable parameters. For example, imagine a batch normalization layer that tracks running statistics (like running mean and variance) during training. These values need to:

  • Move with the model when you call .to(device)
  • Be included when you save the model
  • Not be updated by the optimizer

If you just assign these as regular Python attributes (self.running_mean = torch.zeros(10)), they won't automatically move devices with your model!

What is register_buffer()?

The register_buffer() method tells PyTorch: "This tensor is part of my model's state, but it's not a learnable parameter." It's like having a notebook that travels with your model everywhere, but the optimizer never writes in it.

Analogy: Think of parameters as the model's adjustable knobs (learned through backpropagation), while buffers are its fixed gauges or displays—they show information and need to stay with the model, but they're not turned during training.

When to Use Buffers

Common use cases:

  • Running statistics in normalization layers
  • Fixed positional encodings
  • Precomputed masks or constants
  • Counters or iteration trackers

Buffers are registered in __init__() and automatically handled by PyTorch's device management system.

Key Takeaway: Use register_buffer() to store non-learnable tensors that should move with your model across devices and be included in the model's state, distinguishing them from trainable parameters.