BiteSizedChunks.comLearn one small thing at a time.

Course contentsShow

589The Biological Neuron: Inspiration for Artificial Networks
590The Perceptron: A Single Artificial Neuron
591Perceptron Learning Rule: Training a Single Neuron
592Perceptron Limitations: The XOR Problem
593From Step to Continuous: Introducing Activation Functions
594The Multilayer Perceptron: Stacking Layers
595Why Hidden Layers Matter: Universal Approximation
596Network Architecture Terminology: Depth and Width
597Fully Connected Layers: Dense Connections
598Matrix Representation of Layer Computations
599The Need for Nonlinearity: What Happens Without It
600Depth vs Width: Architectural Trade-offs
601From Two-Layer to Deep Networks
602Representing Classification and Regression Outputs
603What Forward Propagation Computes
604Single Neuron Forward Pass
605Layer-by-Layer Computation
606Matrix Formulation of Forward Pass
607Batched Forward Propagation
608Intermediate Activations and Their Role
609Forward Pass Through Multi-Layer Networks
610Forward Propagation in Different Architectures
611Numerical Stability in Forward Pass
612Implementing Forward Propagation from Scratch
613Loss Functions: Purpose and Role in Training
614Mean Squared Error for Regression
615Mean Absolute Error and Huber Loss
616Binary Cross-Entropy Loss
617Categorical Cross-Entropy Loss
618Sparse Categorical Cross-Entropy
619Cross-Entropy Mathematics and Information Theory
620Focal Loss for Class Imbalance
621Hinge Loss and Margin-Based Losses
622Contrastive and Triplet Losses
623Loss Function Choice and Task Alignment
624Custom Loss Functions and Multi-Task Learning
625The Chain Rule: Foundation of Backpropagation
626Computational Graph Representation
627Forward Pass: Computing Activations Layer by Layer
628Loss Function Gradient: Starting Backpropagation
629Output Layer Gradient Derivation
630Hidden Layer Gradient: Recursive Application
631Gradient of Common Activation Functions
632Matrix Form Backpropagation
633Backpropagation for Fully Connected Layers
634The Backward Pass Algorithm
635Jacobian Matrices in Backpropagation
636Backpropagation Through Time: RNN Preview
637Numerical Gradient Checking
638Memory Requirements of Backpropagation
639Common Backpropagation Implementation Mistakes
640From Theory to Code: Implementing Backpropagation
641What is a Computational Graph?
642Forward Pass Through a Computational Graph
643The Chain Rule in Computational Graphs
644Backward Pass and Gradient Accumulation
645Automatic Differentiation Fundamentals
646Forward Mode vs Reverse Mode Autodiff
647Dynamic vs Static Computational Graphs
648Tracking Operations for Gradient Computation
649Gradient Checkpointing and Memory Trade-offs
650Detaching Tensors and Stopping Gradients

589The Biological Neuron: Inspiration for Artificial Networks
590The Perceptron: A Single Artificial Neuron
591Perceptron Learning Rule: Training a Single Neuron
592Perceptron Limitations: The XOR Problem
593From Step to Continuous: Introducing Activation Functions
594The Multilayer Perceptron: Stacking Layers
595Why Hidden Layers Matter: Universal Approximation
596Network Architecture Terminology: Depth and Width
597Fully Connected Layers: Dense Connections
598Matrix Representation of Layer Computations
599The Need for Nonlinearity: What Happens Without It
600Depth vs Width: Architectural Trade-offs
601From Two-Layer to Deep Networks
602Representing Classification and Regression Outputs
603What Forward Propagation Computes
604Single Neuron Forward Pass
605Layer-by-Layer Computation
606Matrix Formulation of Forward Pass
607Batched Forward Propagation
608Intermediate Activations and Their Role
609Forward Pass Through Multi-Layer Networks
610Forward Propagation in Different Architectures
611Numerical Stability in Forward Pass
612Implementing Forward Propagation from Scratch
613Loss Functions: Purpose and Role in Training
614Mean Squared Error for Regression
615Mean Absolute Error and Huber Loss
616Binary Cross-Entropy Loss
617Categorical Cross-Entropy Loss
618Sparse Categorical Cross-Entropy
619Cross-Entropy Mathematics and Information Theory
620Focal Loss for Class Imbalance
621Hinge Loss and Margin-Based Losses
622Contrastive and Triplet Losses
623Loss Function Choice and Task Alignment
624Custom Loss Functions and Multi-Task Learning
625The Chain Rule: Foundation of Backpropagation
626Computational Graph Representation
627Forward Pass: Computing Activations Layer by Layer
628Loss Function Gradient: Starting Backpropagation
629Output Layer Gradient Derivation
630Hidden Layer Gradient: Recursive Application
631Gradient of Common Activation Functions
632Matrix Form Backpropagation
633Backpropagation for Fully Connected Layers
634The Backward Pass Algorithm
635Jacobian Matrices in Backpropagation
636Backpropagation Through Time: RNN Preview
637Numerical Gradient Checking
638Memory Requirements of Backpropagation
639Common Backpropagation Implementation Mistakes
640From Theory to Code: Implementing Backpropagation
641What is a Computational Graph?
642Forward Pass Through a Computational Graph
643The Chain Rule in Computational Graphs
644Backward Pass and Gradient Accumulation
645Automatic Differentiation Fundamentals
646Forward Mode vs Reverse Mode Autodiff
647Dynamic vs Static Computational Graphs
648Tracking Operations for Gradient Computation
649Gradient Checkpointing and Memory Trade-offs
650Detaching Tensors and Stopping Gradients

Lesson 619 of 3,538·15. Neural Network FoundationsPro lesson

Cross-Entropy Mathematics and Information Theory

Understanding the information-theoretic foundation of cross-entropy as KL divergence between target and predicted distributions.

This lesson is for subscribers

You've completed the free preview. Subscribe to unlock every lesson in every course.

See pricing Back to course