BiteSizedChunks.comLearn one small thing at a time.

Course contentsShow

Machine Learning and Deep Learning

775What is a Tensor?
776Creating Tensors from Data
777Tensor Initialization Functions
778Tensor Attributes: Shape, Dtype, and Device
779Indexing and Slicing Tensors
780Reshaping Tensors: View and Reshape
781Element-wise Operations
782Broadcasting Mechanics
783Matrix Multiplication and Linear Algebra
784Reduction Operations
785Tensor Concatenation and Stacking
786In-place Operations and Memory
787Type Casting and Conversions
788Common Tensor Pitfalls and Best Practices
789What is Autograd and Why It Matters
790The requires_grad Flag
791The Computational Graph
792Computing Gradients with .backward()
793Accessing Gradients with .grad
794Gradient Accumulation and .zero_grad()
795Detaching Tensors from the Graph
796The torch.no_grad() Context Manager
797Non-Scalar Outputs and Gradient Arguments
798Gradient Function and the grad_fn Attribute
799Higher-Order Derivatives
800Autograd Profiling and Common Pitfalls
801Understanding nn.Module: The Base Class for All Models
802Creating Your First Custom Module
803The Forward Method: Defining Model Computation
804Automatic Parameter Registration
805Common Layer Types: Linear, Conv2d, and More
806Sequential Containers for Simple Architectures
807ModuleList and ModuleDict for Dynamic Architectures
808Nested Modules: Building Blocks and Composition
809Accessing and Iterating Over Parameters
810Training vs Evaluation Mode: model.train() and model.eval()
811Moving Models Between Devices: CPU and GPU
812Registering Buffers for Non-Learnable State
813Hooks: Intercepting Forward and Backward Passes
814Model Surgery: Freezing Layers and Transfer Learning Setup
815PyTorch Datasets: Creating Custom Dataset Classes
816Built-in Datasets and torchvision.datasets
817DataLoader Fundamentals: Batching and Shuffling
818Collate Functions: Custom Batch Creation
819num_workers: Multiprocess Data Loading
820pin_memory and GPU Transfer Optimization
821Transforms and Data Preprocessing Pipelines
822Samplers: Controlling Data Access Patterns
823Distributed Samplers for Multi-GPU Training
824Iterable Datasets for Streaming Data
825Prefetching and DataLoader Performance Tuning
826Handling Imbalanced Data in DataLoaders
827Anatomy of a Training Loop
828Training vs Evaluation Mode
829Zero Gradients and Gradient Accumulation
830Validation Loop Implementation
831Loss and Metric Tracking
832Early Stopping Implementation
833Learning Rate Scheduling
834Checkpointing: Saving Model State
835Loading and Resuming Training
836Best Model Tracking
837Training Progress Monitoring
838Reproducibility: Random Seeds
839Mixed Precision Training Basics
840Training Loop Best Practices
841CPU vs GPU Architecture for Deep Learning
842Checking CUDA Availability in PyTorch
843Moving Tensors to GPU with .to() and .cuda()
844Device Management Best Practices
845Moving Models to GPU
846GPU Memory Management Fundamentals
847Monitoring GPU Memory Usage
848Common GPU Memory Pitfalls
849Multi-GPU Basics: DataParallel
850Optimizing CPU-GPU Data Transfer

Machine Learning and Deep Learning

775What is a Tensor?
776Creating Tensors from Data
777Tensor Initialization Functions
778Tensor Attributes: Shape, Dtype, and Device
779Indexing and Slicing Tensors
780Reshaping Tensors: View and Reshape
781Element-wise Operations
782Broadcasting Mechanics
783Matrix Multiplication and Linear Algebra
784Reduction Operations
785Tensor Concatenation and Stacking
786In-place Operations and Memory
787Type Casting and Conversions
788Common Tensor Pitfalls and Best Practices
789What is Autograd and Why It Matters
790The requires_grad Flag
791The Computational Graph
792Computing Gradients with .backward()
793Accessing Gradients with .grad
794Gradient Accumulation and .zero_grad()
795Detaching Tensors from the Graph
796The torch.no_grad() Context Manager
797Non-Scalar Outputs and Gradient Arguments
798Gradient Function and the grad_fn Attribute
799Higher-Order Derivatives
800Autograd Profiling and Common Pitfalls
801Understanding nn.Module: The Base Class for All Models
802Creating Your First Custom Module
803The Forward Method: Defining Model Computation
804Automatic Parameter Registration
805Common Layer Types: Linear, Conv2d, and More
806Sequential Containers for Simple Architectures
807ModuleList and ModuleDict for Dynamic Architectures
808Nested Modules: Building Blocks and Composition
809Accessing and Iterating Over Parameters
810Training vs Evaluation Mode: model.train() and model.eval()
811Moving Models Between Devices: CPU and GPU
812Registering Buffers for Non-Learnable State
813Hooks: Intercepting Forward and Backward Passes
814Model Surgery: Freezing Layers and Transfer Learning Setup
815PyTorch Datasets: Creating Custom Dataset Classes
816Built-in Datasets and torchvision.datasets
817DataLoader Fundamentals: Batching and Shuffling
818Collate Functions: Custom Batch Creation
819num_workers: Multiprocess Data Loading
820pin_memory and GPU Transfer Optimization
821Transforms and Data Preprocessing Pipelines
822Samplers: Controlling Data Access Patterns
823Distributed Samplers for Multi-GPU Training
824Iterable Datasets for Streaming Data
825Prefetching and DataLoader Performance Tuning
826Handling Imbalanced Data in DataLoaders
827Anatomy of a Training Loop
828Training vs Evaluation Mode
829Zero Gradients and Gradient Accumulation
830Validation Loop Implementation
831Loss and Metric Tracking
832Early Stopping Implementation
833Learning Rate Scheduling
834Checkpointing: Saving Model State
835Loading and Resuming Training
836Best Model Tracking
837Training Progress Monitoring
838Reproducibility: Random Seeds
839Mixed Precision Training Basics
840Training Loop Best Practices
841CPU vs GPU Architecture for Deep Learning
842Checking CUDA Availability in PyTorch
843Moving Tensors to GPU with .to() and .cuda()
844Device Management Best Practices
845Moving Models to GPU
846GPU Memory Management Fundamentals
847Monitoring GPU Memory Usage
848Common GPU Memory Pitfalls
849Multi-GPU Basics: DataParallel
850Optimizing CPU-GPU Data Transfer

← Machine Learning and Deep Learning

Lesson 794 of 3,538·19. PyTorch FundamentalsPro lesson

Gradient Accumulation and .zero_grad()

Why gradients accumulate by default and how to clear them between training iterations using zero_grad().

This lesson is for subscribers

You've completed the free preview. Subscribe to unlock every lesson in every course.

See pricing Back to course