Machine Learning and Deep Learning Glossary
Key terms from the Machine Learning and Deep Learning course, linked to the lesson that introduces each one.
972 terms.
A
- ablation study
- An ablation study is like reverse engineering a recipe: you start with a complex dish (Rainbow DQN with all improvements) and remove ingredients one at a time to see which ones actually matter.
- Lesson 2236 — Ablation Studies: Which Improvements Matter Most
- Above the line
- Model is underconfident (predicts lower probabilities than observed)
- Lesson 489 — Calibration Plots and Reliability DiagramsLesson 530 — Reliability Diagrams
- Absolute scoring
- Absolute scoring is efficient—one evaluation per output—and provides interpretable numeric feedback.
- Lesson 3162 — Pairwise Comparison vs Absolute Scoring
- Accuracy becomes misleading
- High accuracy doesn't mean your model is actually useful
- Lesson 242 — Class Imbalance Introduction
- Accuracy Loss
- Accuracy Loss is your usual objective (cross-entropy, MSE, etc.)
- Lesson 3310 — Fairness Constraints During Training
- action space
- An action space is the complete set of operations an agent can perform—its "toolbox." A tool registry is the system that catalogs these tools with structured descriptions so the agent knows what's available and how to use each one.
- Lesson 2062 — Action Space and Tool RegistryLesson 2134 — States, Actions, and State Spaces
- actions
- Your actions are driving to connected cities or refueling.
- Lesson 2083 — Planning in AI Agents: Problem Formulation
- activation
- Each layer's output is called an activation or intermediate activation.
- Lesson 609 — Forward Pass Through Multi-Layer Networks
- Activation atlases
- Imagine exploring a vast museum where every room represents what a neural network layer "sees." Activation atlases are exactly that—comprehensive maps of learned representations created by collecting millions of neuron activations, clustering them by similarity, and visualizing what each cluster represents.
- Lesson 3272 — Activation Atlases and Feature Spaces
- Actor
- In actor-critic methods, the actor is a neural network that directly represents your policy—it's the component that decides what action to take given a state.
- Lesson 2275 — From Pure Policy Gradients to Actor-CriticLesson 2277 — The Actor: Parameterized Policy Networks
- Acts as regularization
- the batch statistics add noise during training (similar to dropout's effect)
- Lesson 752 — Batch Normalization: Core Concept
- Acyclic
- Acyclic means no circular dependencies—you can't have Task A depending on Task B, which depends on Task C, which depends back on Task A
- Lesson 2861 — Directed Acyclic Graphs (DAGs)
- AdaBound
- AdaMax and AdaBound are two clever variants that address specific limitations of standard Adam.
- Lesson 709 — AdaMax and AdaBound Variants
- additive smoothing
- Laplace smoothing (also called additive smoothing) adds a small "pseudocount" to every possible feature-class combination, even those you've never observed.
- Lesson 334 — Laplace Smoothing for Zero Probabilities
- adjacency matrix
- The adjacency matrix is one fundamental representation: a square matrix where rows and columns represent nodes, and cell values indicate whether an edge exists between them.
- Lesson 2484 — Graph Representations: Adjacency Matrix
- Adjust learning rates
- If gradients are consistently large or small, tune accordingly
- Lesson 680 — Gradient Norm Monitoring
- Advantage
- Errors are more balanced—sometimes slightly high, sometimes slightly low—so they tend to cancel out rather than compound. This usually preserves model accuracy better.
- Lesson 2627 — Quantization Error and Rounding
- advantage function
- The advantage function is a clever mathematical trick that tells us how much better (or worse) an action is compared to the average action we'd take in that state.
- Lesson 2278 — Advantage Functions in Actor-Critic
- adversarial patches
- Unlike typical adversarial examples that apply tiny perturbations across an entire image, adversarial patches are localized regions—think stickers or decals—that can be placed anywhere in a scene to cause misclassification.
- Lesson 3385 — Adversarial PatchesLesson 3394 — Adversarial Patches
- agents
- Understanding the distinction between chains, workflows, and agents is crucial for choosing the right architecture.
- Lesson 2058 — Agent vs. Chain vs. WorkflowLesson 2876 — Prefect Cloud and Deployment Patterns
- AI agent
- An AI agent is a system that operates with a degree of autonomy—it observes its environment, makes decisions based on those observations, and takes actions to accomplish specific objectives.
- Lesson 2057 — What is an AI Agent?
- AI alignment problem
- The AI alignment problem is the challenge of ensuring that AI systems pursue the goals and values their designers intend, rather than unintended interpretations or proxy metrics that can lead to harmful outcomes.
- Lesson 3425 — What is the AI Alignment Problem?
- AlexNet
- In 2012, a team led by Alex Krizhevsky submitted a convolutional neural network called AlexNet to the ImageNet Large Scale Visual Recognition Challenge (ILSVRC).
- Lesson 890 — AlexNet: The Deep Learning Revolution
- ALiBi takes a different approach
- instead of adding positional information to token embeddings, it adds a linear penalty directly to the attention scores based on the distance between tokens. The further apart two tokens are, the more negative bias is added to their attention score, making distant tokens slightly less likely to attend strongly to each other.
- Lesson 1656 — ALiBi: Attention with Linear Biases
- Analogy
- Think of a car wash with four stations. With one car, three stations are always idle. But if you send four cars through in a row (microbatching), all stations stay busy most of the time.
- Lesson 39 — Higher-Order DerivativesLesson 149 — NumPy Arrays vs Python Lists for MLLesson 317 — LightGBM: Gradient Boosting with Histogram-Based SplittingLesson 955 — R-CNN ArchitectureLesson 1360 — Using Hierarchical Features for DetectionLesson 1522 — GAN Evaluation Beyond Inception ScoreLesson 2625 — The Quantization Equation and DequantizationLesson 2661 — Activation Quantization ChallengesLesson 2732 — All-Gather and Reduce-Scatter OperationsLesson 2757 — GPipe: Microbatching and Pipeline Bubbles
- anomaly
- An anomaly is a broader concept: any pattern that doesn't conform to expected behavior, even if individual values seem reasonable (like a user profile with age=25 but retirement_status=true).
- Lesson 3056 — Outlier and Anomaly Detection in Data
- Anomaly Detection
- Since autoencoders learn to reconstruct normal training data well, they struggle with unusual inputs. If reconstruction loss is unusually high for a new sample, it's likely an anomaly. This works brilliantly for fraud detection, manufacturing defect identification, or medical imaging anomalies—the autoencoder simply hasn't "seen" these patterns before.
- Lesson 1440 — Applications and Limitations of Basic Autoencoders
- ANOVA F-statistic
- Tests if feature means differ significantly across target classes
- Lesson 444 — Feature Selection: Filter Methods
- Answer distributions
- Most datasets have imbalanced answer frequencies (e.g., "yes" and "no" are very common)
- Lesson 1409 — Visual Question Answering Task Definition
- Arbitration agent
- A higher-level agent (from hierarchical architectures, lesson 2115) makes the final call
- Lesson 2116 — Consensus and Voting Mechanisms
- Arguments
- A dictionary or JSON object with parameter names and values (e.g., {"location": "Paris", "unit": "celsius"})
- Lesson 1925 — Parsing Function Call Responses
- Assistant
- Think of it like a play: the system is the director's notes (invisible to the audience but guiding everything), the user is the audience asking questions, and the assistant is the actor performing on stage.
- Lesson 1232 — Instruction Format and Template DesignLesson 1752 — Instruction Format and TemplatesLesson 1854 — System vs User vs Assistant Messages
- Asymmetric models
- Asymmetric models are optimized for query-document pairs with different characteristics.
- Lesson 1974 — Asymmetric vs Symmetric Retrieval
- Asymmetric retrieval
- Asymmetric retrieval is what happens in typical search scenarios: you have a short, incomplete query (like "best pizza recipes") and need to find relevant documents (full recipe articles).
- Lesson 1974 — Asymmetric vs Symmetric Retrieval
- Asynchronous training
- Asynchronous training is more like independent study—workers compute gradients and immediately update a shared parameter server without waiting for others.
- Lesson 2708 — Synchronous vs Asynchronous Training
- Atrous Spatial Pyramid Pooling
- DeepLab often combines multiple atrous rates in parallel (called Atrous Spatial Pyramid Pooling) to capture objects at different scales simultaneously.
- Lesson 981 — DeepLab and Atrous Convolutions
- Attack Success Rate
- Attack Success Rate is your primary metric.
- Lesson 3336 — Measuring Privacy Leakage Empirically
- Attention rollout
- Attention rollout is a technique that combines attention weights across all layers to create a single attention map showing how input tokens influence the final representation.
- Lesson 3259 — Attention Rollout and Flow
- attention weights
- The resulting values α₁, α₂, ..., αₙ are called attention weights.
- Lesson 1041 — Softmax Normalization and Attention Weights
- AUC
- The ROC curve (Receiver Operating Characteristic) and its AUC (Area Under Curve) are popular, but they can be overly optimistic for imbalanced data.
- Lesson 379 — Evaluation Metrics for Anomaly Detection
- Audio Source Separation
- Audio Source Separation is the task of taking a mixed audio signal and separating it back into its constituent sources.
- Lesson 2481 — Audio Source Separation
- auto wrap policy
- An auto wrap policy is a set of rules that tells FSDP: "wrap modules that meet these criteria." PyTorch FSDP provides two main policies:
- Lesson 2736 — Auto Wrap Policy in PyTorch FSDP
- autoencoder
- An autoencoder is an unsupervised neural network that learns to compress data into a compact representation and then reconstruct the original input from that compressed form.
- Lesson 378 — Autoencoders for Anomaly DetectionLesson 406 — Autoencoders for Dimensionality ReductionLesson 1429 — What Autoencoders Are and Why They Matter
- autoencoder's decoder
- The autoencoder's decoder is the bridge that transforms latent representations back into visual reality.
- Lesson 1576 — Decoder Consistency and Reconstruction Quality
- autoregressive generation
- This is called autoregressive generation: each output becomes the next input, creating a chain of predictions that builds the complete sequence.
- Lesson 1030 — Inference and Autoregressive Generation
- Autoregressive inference
- Autoregressive inference means the decoder generates output sequentially: it produces one token, then uses that token as input to generate the next token, then uses both previous tokens to generate the third, and so on.
- Lesson 1100 — Autoregressive Inference
- autoregressive sampling
- This is called autoregressive sampling because each step depends on (regresses on) the model's own previous outputs.
- Lesson 1190 — Autoregressive Sampling at Inference
- Average Precision
- For a single user's recommendation list, Average Precision is the mean of precision values calculated at each position where a relevant item appears.
- Lesson 2376 — Mean Average Precision (MAP)
B
- Backfilling
- Backfilling is computing features for historical data, typically when you:
- Lesson 2887 — Feature Materialization and Backfilling
- Backtracking
- Backtracking means returning to an earlier decision point (a parent node in the tree) to try a different path.
- Lesson 1894 — Backtracking and Path Refinement
- Backward fill
- Backward fill is useful when you know the sequence will continue and want to use future information (e.g., filling hourly data when you have the next hour's reading).
- Lesson 433 — Forward Fill and Backward Fill for Time Series
- Barriers
- Barriers are synchronization points where all processes must "wait" until everyone arrives before continuing.
- Lesson 2797 — Synchronization and Barrier Operations
- base model
- A base model is the raw pretrained transformer that has learned language patterns from massive text corpora through next-token prediction.
- Lesson 1227 — Base Models: Pretraining Objective and CapabilitiesLesson 1750 — Base Models vs Instruction-Tuned Models
- Base models
- Base models are like blank canvases—they predict what comes next based on patterns, excellent for raw completion
- Lesson 1233 — When to Use Base vs Instruction-Tuned Models
- Base weights
- Base weights are stored in low precision (4-bit or 8-bit)
- Lesson 1725 — Quantization Basics for Fine-Tuning
- baseline
- A baseline is any function b(s) that depends only on the state (not the action).
- Lesson 2256 — Baselines for Variance Reduction
- baseline model
- A baseline model is an intentionally simple approach that establishes a minimum performance bar.
- Lesson 141 — Baseline Models: Starting Simple
- basis
- In linear algebra, a basis is a set of linearly independent vectors that span the entire vector space.
- Lesson 11 — Basis and Dimension
- batch
- Instead of computing predictions one-by-one in a loop, we organize multiple inputs as rows in a matrix (called a batch).
- Lesson 607 — Batched Forward Propagation
- batch learning
- Think of it like learning to cook: batch learning is like taking a cooking class once a year where you practice 100 recipes together.
- Lesson 132 — Online Learning: Updating Models in Real-Time
- Batch Normalization
- Contains running statistics and scale/shift parameters that are sensitive to precision loss; often fused with preceding conv layers instead
- Lesson 2641 — Quantization of Specific Layer Types
- Batch normalization layers
- Biases are typically initialized to zero, but the scale parameter may start at one
- Lesson 671 — Bias Initialization
- Batch size
- Larger batches (256+) are common; scale LR proportionally
- Lesson 913 — Residual Networks in Practice
- Batch-aware caching
- Batch-aware caching is the strategy of separating cached from uncached requests, processing only what's necessary, and reassembling the full batch response in the correct order.
- Lesson 2923 — Batch-Aware Caching
- Batched forward propagation
- Batched forward propagation means stacking multiple input samples together and processing them all simultaneously through the same matrix operations.
- Lesson 607 — Batched Forward Propagation
- Bayesian approach
- Instead of one fixed value, you maintain a distribution over possible parameter values. Rather than saying "the weight is 0.73," you say "the weight is probably around 0.73, with this much uncertainty." You get something like "weight ~ Normal(0.73, 0.15)" where the spread represents your confidence.
- Lesson 557 — From Frequentist to Bayesian Perspective
- Beam A's page table
- Beam A's page table is updated to point to the new page; beam B keeps using the shared one
- Lesson 2974 — Copy-on-Write for Shared Prefixes
- Beam Width Selection
- Typical values are 3-10. Larger beams find better captions but cost more computation. Empirically, quality plateaus around width 5-7 for captioning.
- Lesson 1407 — Beam Search for Caption Generation
- Bellman backup
- A Bellman backup is the fundamental operation that updates a value estimate at a state (or state-action pair) by looking one step ahead and combining immediate reward with discounted future values.
- Lesson 2156 — Bellman Backup Operations
- Bellman Expectation Equation
- The Bellman Expectation Equation is a fundamental recursive relationship that breaks down the value function V(s) into two components:
- Lesson 2149 — The Bellman Expectation Equation for V
- Below the line
- Model is overconfident (predicts higher probabilities than observed)
- Lesson 489 — Calibration Plots and Reliability DiagramsLesson 530 — Reliability Diagrams
- Bernoulli distribution
- The Bernoulli distribution describes this random variable with one parameter p (the probability of success).
- Lesson 64 — Common Discrete Distributions: Bernoulli and Binomial
- Bernoulli Naive Bayes
- Calculate the proportion of documents where each feature is present/absent per class
- Lesson 335 — Training Naive Bayes: Parameter Estimation
- BERTviz
- BERTviz is the most popular library for attention visualization.
- Lesson 3261 — Attention Visualization Tools and Libraries
- Best practices
- Run pilot annotations, measure kappa early, refine guidelines when agreement is low, and use gold standard examples to maintain consistency across annotators.
- Lesson 3178 — Annotation Quality and Inter-Rater Agreement
- Beta-Binomial conjugacy
- If you have a Beta prior on probability and observe Binomial data (coin flips), the posterior is also Beta
- Lesson 580 — Conjugate Priors and Analytical Posteriors
- Better attention
- The cross-attention mechanism lets each word directly query any image patch, just like the visual attention mechanisms you learned, but more flexible.
- Lesson 1408 — Transformer-Based Image Captioning
- Better disambiguation
- Words with multiple meanings are easier to understand with full context
- Lesson 1186 — Left-to-Right vs Bidirectional Context
- Better final convergence
- The gentle final approach helps find better local minima
- Lesson 717 — Cosine Annealing
- Better generalization
- The model learns relationships independent of absolute position
- Lesson 1087 — Relative Positional Encodings in Transformers
- Better learning
- The model focuses on high-level structure, not pixel noise
- Lesson 1567 — Latent Space Properties and Dimensionality
- Better Memory Utilization
- Traditional serving pre-allocates contiguous memory for the full KV cache, wasting space when sequences vary in length. vLLM's paged attention uses memory blocks on-demand, like an operating system manages RAM. This efficiency means you can fit more concurrent requests in the same GPU memory budget.
- Lesson 2979 — Performance Characteristics of vLLM
- Better semantic integrity
- Each chunk is more likely to be self-contained and meaningful
- Lesson 1987 — Paragraph-Based Chunking
- Bi-encoder
- A bi-encoder is an embedding model architecture that uses two independent neural networks (or the same network twice) to convert text into dense vector representations.
- Lesson 1334 — Late Interaction Models (ColBERT)Lesson 1951 — Embedding Models: Bi-Encoders for Retrieval
- BiLSTM handles local context
- By processing text bidirectionally, it captures rich features about each token based on surrounding words. For "Apple Inc.", it learns that "Apple" here is business-related, not fruit-related.
- Lesson 1291 — BiLSTM-CRF Architecture for NER
- Binary Relevance
- Binary Relevance is the simplest approach to handle this: you create a separate yes/no classifier for each label.
- Lesson 550 — Problem Transformation: Binary Relevance
- BIOES scheme
- The BIOES scheme is more explicit with five tags:
- Lesson 1288 — NER Tag Schemes: IOB and BIOES
- Blackboard architecture
- A shared workspace where agents post findings that others can read
- Lesson 2120 — Shared Context and Memory in Multi-Agent Systems
- Block patterns
- The model groups related concepts together, showing it understands phrase boundaries or semantic clusters.
- Lesson 1059 — Understanding Attention Weight Visualization
- Block-Level Wrapping
- Wrap logical modules (e.g., entire Transformer blocks containing attention + FFN). Balances memory and communication—parameters for the whole block are gathered once, used for multiple operations, then discarded.
- Lesson 2735 — Unit vs Full Shard Wrapping Strategies
- Blurriness
- The decoder averages out fine details it cannot precisely reconstruct
- Lesson 1576 — Decoder Consistency and Reconstruction Quality
- BM25
- BM25 (Best Match 25) is a probabilistic ranking function used in traditional information retrieval.
- Lesson 1998 — Keyword Search Fundamentals: BM25
- Bootstrapping
- Bootstrapping means the critic doesn't need complete trajectories.
- Lesson 2280 — Temporal Difference Learning in the Critic
- Both simultaneously
- The most challenging scenario requiring retraining and data collection
- Lesson 3041 — Concept Drift vs Data Drift
- bottleneck
- In an autoencoder, the bottleneck is the narrowest layer—typically in the middle—where your data gets squeezed into far fewer dimensions than the original input.
- Lesson 677 — Gradient Flow Analysis Through Network DepthLesson 1431 — The Bottleneck and Latent Space
- Business utility
- A 1-hour-ahead forecast serves different needs than a 1-month-ahead forecast
- Lesson 2395 — Forecasting Horizon and Evaluation Windows
C
- C hyperparameter
- The C hyperparameter is your control knob for this trade-off.
- Lesson 273 — The C Hyperparameter
- Caching layers
- Caching layers are empty (KV cache blocks, result caches)
- Lesson 3009 — Model Warmup and Cold Start Optimization
- Calibration
- Calibration refers to how well the reward model's confidence matches actual quality.
- Lesson 1784 — Calibration and Score Distributions
- canary deployment
- A canary deployment is a progressive rollout strategy where you route a small percentage of production traffic (e.g., 5%) to your new model while the majority continues using the existing model.
- Lesson 3084 — Canary Deployment
- Captures non-linearity
- A linear model can now treat different age ranges differently without polynomial features
- Lesson 441 — Binning and Discretization Techniques
- Carbon-aware scheduling
- Carbon-aware scheduling means timing your model training to run when the grid is cleanest.
- Lesson 3472 — Carbon-Aware Training and Scheduling
- cardinality
- ResNeXt extends ResNet by introducing a new dimension called cardinality — the number of parallel transformation paths within a block.
- Lesson 912 — ResNeXt: Aggregated Residual Transformations
- Catastrophic forgetting
- The model might optimize for reward but lose its grammar, coherence, or factual knowledge
- Lesson 1791 — The Trust Region Constraint
- CatBoost
- CatBoost is often the slowest during training because it handles categorical features natively with more sophisticated preprocessing.
- Lesson 320 — Comparing Boosting Libraries: XGBoost vs LightGBM vs CatBoost
- Categorical Cross-Entropy
- Categorical Cross-Entropy is its natural extension to multiple classes (3 or more).
- Lesson 617 — Categorical Cross-Entropy Loss
- Causal
- The model only looks backward in time (never into the future), essential for real-time generation
- Lesson 2468 — Neural Vocoders: WaveNet
- Centering
- The teacher's outputs are normalized by subtracting a running average of previous batch centers. Think of it like preventing the model from always predicting the same "safe" answer by shifting the origin dynamically.
- Lesson 2567 — DINO: Self-Distillation with No Labels
- Centralized store
- A single vector database or knowledge graph all agents query and update
- Lesson 2120 — Shared Context and Memory in Multi-Agent Systems
- chain
- A chain is a predetermined sequence of steps executed in order, like a recipe.
- Lesson 2058 — Agent vs. Chain vs. Workflow
- Channel shuffle
- Channel shuffle is an elegant operation that mixes information across groups without expensive computation.
- Lesson 923 — ShuffleNet: Channel Shuffle Operations
- Character-level
- Character-level is more human-intuitive but fragile with unexpected inputs.
- Lesson 1644 — Byte-Level vs Character-Level Tokenization
- characteristic equation
- The characteristic equation is our main tool.
- Lesson 17 — Computing Eigenvalues and Eigenvectors
- Circuits
- Circuits are how features connect and compute together.
- Lesson 3266 — Circuits vs Features in Neural Networks
- Class 0
- Often called the "negative class" (e.g., not spam, disease absent, transaction legitimate)
- Lesson 236 — Binary Classification Setup
- Class 1
- Often called the "positive class" (e.g., spam, disease present, transaction fraudulent)
- Lesson 236 — Binary Classification Setup
- Class imbalance effects
- 99% accuracy means nothing if your model just predicts "negative" for everything in a 99:1 imbalanced dataset
- Lesson 3128 — Why Aggregate Metrics Hide Problems
- Classification
- Predict a category → "This email is spam" or "This tumor is benign"
- Lesson 235 — What is Classification?
- Classification and Escalation
- Not every anomaly is an incident. Define severity levels (e.g., critical: immediate harm to users; moderate: degraded performance; low: edge case failures) and clear escalation paths to appropriate stakeholders—technical teams, legal, communications, ethics boards.
- Lesson 3535 — Incident Response and Management
- Clear preference signal
- The chosen response should be meaningfully better than the rejected one. Ambiguous pairs where both are equally good or bad provide weak learning signal.
- Lesson 1810 — Preference Dataset Requirements for DPO
- CLIP as a vision encoder
- CLIP as a vision encoder is particularly powerful because it was pretrained to align image and text representations in the same embedding space.
- Lesson 1416 — Vision Encoders for Multimodal LLMs
- Clustering
- Clustering is a core unsupervised learning technique that groups similar data points together based on their features alone.
- Lesson 337 — What is Clustering?
- CNNs
- CNNs are great at local feature extraction but struggle with long-range dependencies
- Lesson 2457 — Conformer Architecture for ASR
- collate function
- A collate function is a custom function you pass to DataLoader that controls exactly how individual samples get combined into a batch.
- Lesson 818 — Collate Functions: Custom Batch Creation
- column
- Each column represents one key position (what's being "looked at")
- Lesson 1053 — The Attention Score Matrix
- column space
- Remember that the column space is all possible linear combinations of a matrix's columns.
- Lesson 13 — Rank of a Matrix
- component
- In Kubeflow, a component is a self-contained unit of work—like preprocessing data, training a model, or evaluating results.
- Lesson 2878 — Building Kubeflow Pipeline Components
- Components
- Each Gaussian distribution (you learned this in "Gaussian Distribution as Cluster Model") represents one "ingredient"
- Lesson 365 — Mixture Model Definition
- Compounding errors
- As models are trained on tasks we can't fully verify, small misalignments may amplify over time
- Lesson 3431 — The Scalable Oversight Problem
- Comprehensive evaluation
- Comprehensive evaluation means tracking the full constellation of metrics—not just optimizing for one—and ensuring your intervention is a net positive across fairness, accuracy, and other operational constraints.
- Lesson 3316 — Evaluating Mitigation Effectiveness
- Computation cost
- Computing cosine similarity or Euclidean distance across millions of 768-dimensional vectors means billions of floating-point operations
- Lesson 1961 — The Curse of Dimensionality in Vector Search
- Computational efficiency
- Latent representations are typically 4-8× smaller in each dimension. If an image is 512×512 pixels, its latent might be 64×64. That's a 64× reduction in data to process!
- Lesson 287 — Gini Impurity as a Splitting CriterionLesson 607 — Batched Forward PropagationLesson 1354 — Swin Transformer: Hierarchical ArchitectureLesson 1565 — From Pixel Space to Latent Space Diffusion
- computational graph
- A computational graph is PyTorch's behind-the-scenes record of every operation you perform on tensors with requires_grad=True.
- Lesson 641 — What is a Computational Graph?Lesson 791 — The Computational Graph
- Computational Speed
- Mathematical operations are 10-100x faster. When training neural networks with thousands of iterations, this difference means hours versus days.
- Lesson 149 — NumPy Arrays vs Python Lists for ML
- Compute costs
- Computing gradients through backpropagation across all layers is expensive, especially on long sequences.
- Lesson 1711 — The Parameter Efficiency Problem in Fine-Tuning
- Concept drift
- While statistical tests (like those from lesson 3029) can detect data drift, the most direct way to confirm concept drift is watching your model's performance metrics decline over time windows.
- Lesson 3039 — Understanding Concept DriftLesson 3044 — Detecting Concept Drift with Model PerformanceLesson 3047 — Root Cause Analysis for Drift
- conditioning
- This is called conditioning — we're restricting our infinite family of functions to only those that pass through (or near) our observed points.
- Lesson 572 — GP Posterior: Conditioning on DataLesson 579 — Exact Inference: Marginalization and Conditioning
- Confabulated Reasoning
- The model invents plausible-sounding but factually incorrect intermediate steps. Like a student who confidently cites a nonexistent theorem, the model may reference facts, rules, or definitions that don't exist or were misremembered from training data.
- Lesson 1874 — Chain-of-Thought Hallucinations and Errors
- confidence interval
- A confidence interval is a range of values constructed from your sample data that likely contains the true population parameter.
- Lesson 87 — Confidence Intervals
- Confidence-Based Routing
- The model flags low-confidence predictions for human review. If a medical imaging model is 98% confident, it proceeds automatically; at 65% confidence, a radiologist reviews.
- Lesson 3491 — Human-in-the-Loop Design Patterns
- Confusing correlation with causation
- Segment analysis ("model B wins for mobile users!") discovered after testing is exploratory, not confirmatory.
- Lesson 3078 — Interpreting A/B Test Results
- Conjugacy
- Conjugacy means the prior and posterior belong to the same family of distributions.
- Lesson 561 — Conjugate Priors and Analytical Posteriors
- conjugate priors
- But for certain special prior-likelihood pairs, the posterior has the same distributional form as the prior—these are called conjugate priors.
- Lesson 580 — Conjugate Priors and Analytical Posteriors
- Consistency
- The structured thinking process reduces arbitrary judgment variance across similar examples.
- Lesson 3166 — Chain-of-Thought Reasoning for Judges
- Constraints
- Constraints are the rules or limits you must respect while optimizing.
- Lesson 93 — What is Mathematical Optimization?
- Constraints and restrictions
- Constraints and restrictions are explicit rules you embed in your prompt to limit the model's response space and ensure outputs meet your requirements.
- Lesson 1849 — Constraints and Restrictions
- container
- A container is a running instance of that image.
- Lesson 2853 — Docker Containers for ML Projects
- Containment
- Have predefined rollback procedures, model killswitches, or failover to simpler baselines. Sometimes containment means disabling certain features or reverting to human decision-making temporarily.
- Lesson 3535 — Incident Response and Management
- Context awareness
- A recommendation system that assumes high bandwidth and large screens excludes users in low-connectivity regions or those using assistive technologies.
- Lesson 3494 — Inclusive Design and Accessibility
- Context constraints
- Context constraints are your biggest challenge.
- Lesson 1902 — Multi-Step Reasoning Trajectories
- Context encoding
- Context encoding means creating dense vector representations of both the question and potential answer passages.
- Lesson 1301 — Context Encoding and Passage Retrieval
- Contextual routing
- Same query might route to search_vector_db vs. query_sql_database depending on whether the question is about unstructured documents or structured data
- Lesson 2074 — Tool Selection Strategy
- Continued pretraining
- Continued pretraining means taking a pretrained BERT model and running more masked language modeling (MLM) on domain-specific corpora—legal documents, scientific papers, medical records, or financial reports—before your task-specific fine-tuning.
- Lesson 1182 — Domain Adaptation with Continued Pretraining
- Continuous action spaces
- Continuous action spaces are fundamentally different.
- Lesson 2315 — Continuous Action Spaces: Fundamentals
- Continuous auditing
- Continuous auditing means setting up automated systems that regularly recompute the fairness metrics you care about (demographic parity, equalized odds, etc.) and alert you when they degrade beyond acceptable thresholds.
- Lesson 3326 — Continuous Auditing and Monitoring
- Continuous case
- The joint probability density function f(x, y) works similarly, where probabilities come from integrating over regions.
- Lesson 69 — Joint Probability Distributions
- Continuous quality spectrum
- The model learns to denoise across all noise levels—from nearly pure noise to nearly clean images. This comprehensive training coverage ensures it understands the data distribution at multiple scales and levels of detail.
- Lesson 1536 — Why Diffusion Models Generate High Quality
- Continuous risk monitoring
- Continuous risk monitoring means implementing automated systems that constantly evaluate your ML system's health, fairness, security, and alignment with intended use.
- Lesson 3537 — Continuous Risk Monitoring
- Contrastive loss
- Contrastive loss is the key: it's a loss function that explicitly teaches the network to minimize distance between similar pairs (positive pairs) and maximize distance between dissimilar pairs (negative pairs) in the embedding space.
- Lesson 2597 — Contrastive Loss for Siamese Networks
- ControlNet
- ControlNet is an add-on architecture that accepts spatial conditioning signals—images that encode structural information like:
- Lesson 1579 — ControlNet and Spatial Conditioning
- Convergence behavior changes
- The optimization landscape looks "smoother" with less stochastic exploration
- Lesson 2709 — Effective Batch Size in Data Parallelism
- conversation buffer
- A conversation buffer is simply a structured storage of the dialogue history.
- Lesson 2098 — Conversation History Management
- convex function
- A convex function is one where a line segment connecting any two points on its graph lies on or above the curve itself.
- Lesson 97 — Convex Functions
- convex set
- A convex set is a collection of points where, if you pick any two points inside the set, the entire line segment connecting them also lies completely within the set.
- Lesson 96 — Convex Sets
- coordinate ascent
- Mean-field VI optimizes each factor qⱼ(θⱼ) one at a time while holding others fixed—this is called coordinate ascent.
- Lesson 587 — Mean-Field Variational Inference
- Copy-on-Write
- Copy-on-Write is a memory optimization borrowed from operating systems.
- Lesson 2974 — Copy-on-Write for Shared Prefixes
- Corrigibility
- Corrigibility means an AI system remains safely interruptible and modifiable—it cooperates with corrections rather than resisting them.
- Lesson 3435 — Power-Seeking Behavior and Corrigibility
- Cosine
- Cosine is better for high-dimensional vectors where direction matters more than magnitude (like text embeddings).
- Lesson 402 — UMAP: Hyperparameters and Their Effects
- Cost
- Substantial slowdown due to PCIe bandwidth limits. Moving data between CPU and GPU is much slower than GPU-to-GPU communication. Expect 2-5× longer training times.
- Lesson 2737 — CPU Offloading in FSDP
- Cost and Scale
- Hiring qualified annotators is expensive. If you need 100,000 preference pairs to train a robust reward model or DPO policy, you're looking at significant time and budget. Each comparison requires careful reading and judgment.
- Lesson 1817 — Limitations of Human Feedback and Motivation for RLAIF
- covariance matrix
- The covariance matrix is built by applying your kernel function to every pair of points in your dataset.
- Lesson 568 — Kernel Functions and the Covariance Matrix
- Covariates
- Covariates are additional variables that influence your predictions:
- Lesson 2421 — Handling Covariates and External Features
- Crafting Edge Cases
- Red teamers design prompts that sit at the boundary of acceptable behavior—requests that are technically within guidelines but might trigger unsafe outputs. For example, asking for "creative writing about a heist" versus direct instruction to "tell me how to rob a bank."
- Lesson 3449 — Manual Red Teaming Techniques
- CRF enforces global consistency
- The CRF layer looks at the entire sequence of BiLSTM outputs and picks the most coherent label sequence. It prevents impossible transitions like B-PER → I-LOC and ensures entities are properly bounded.
- Lesson 1291 — BiLSTM-CRF Architecture for NER
- Critic
- A value function that estimates how good states or actions are (borrowed from value-based methods like Q-learning)
- Lesson 2275 — From Pure Policy Gradients to Actor-Critic
- critical points
- The critical points are locations where the ground is perfectly flat in all directions—you could place a marble there and it wouldn't roll anywhere.
- Lesson 45 — Critical Points and Extrema
- Critique prompt design
- Critique prompt design is the art of crafting explicit, structured prompts that direct the model's attention toward particular dimensions of quality, making flaws detectable and actionable.
- Lesson 1936 — Critique Prompt Design
- Cron expressions
- Cron expressions are the classic way to define recurring schedules.
- Lesson 2874 — Airflow Scheduling and Triggers
- Cross-attention
- Cross-attention is like looking back at the English text while writing the French translation to find which English words correspond to the French word you're currently generating
- Lesson 1078 — Cross-Attention vs. Self-Attention HeadsLesson 1096 — Cross-Attention Mechanism
- cross-entropy loss
- Log loss, also called cross-entropy loss, measures the quality of probability predictions.
- Lesson 466 — Log Loss (Cross-Entropy Loss)
- cross-feature patterns
- These cross-feature patterns are crucial but hard to hand-engineer.
- Lesson 2368 — Deep Learning for Feature Interactions
- Cross-modal attention
- Cross-modal attention is the bridge that lets one modality query the other.
- Lesson 1376 — Cross-Modal Attention Mechanisms
- Cross-Modal Attention Layers
- Cross-Modal Attention Layers are inserted at regular intervals.
- Lesson 1381 — ViLBERT: Dual-Stream Vision-Language Architecture
D
- DAG
- A DAG is a directed graph with no cycles—you can't follow edges and return to where you started.
- Lesson 2488 — Common Graph Types: Trees, DAGs, and Bipartite Graphs
- Data characteristics
- If input features are predominantly negative, neurons are more vulnerable
- Lesson 655 — The Dying ReLU Problem
- Data diversity
- Data diversity means covering a broad range of tasks, domains, instruction phrasings, and complexity levels.
- Lesson 1755 — Data Quality and Diversity
- data drift
- Input drift (also called data drift or covariate shift) occurs when the statistical distribution of features your model receives in production differs from the distribution it saw during training.
- Lesson 3027 — What is Input Drift and Why It Matters
- Data freshness
- Data freshness refers to how recent your input data is, while latency measures the delay between data generation and availability for inference.
- Lesson 3055 — Freshness and Latency Monitoring
- Data leakage
- Data leakage is the broader phenomenon: any scenario where information from the test set influences training, breaking the fundamental assumption that evaluation data is unseen.
- Lesson 3159 — Benchmark Contamination and Data Leakage
- data parallelism
- Think of it like this: data parallelism is photocopying a book and having multiple readers process different pages simultaneously.
- Lesson 2755 — Model Parallelism vs Data Parallelism
- Data quality
- Data quality refers to how well each instruction-response pair demonstrates the desired behavior.
- Lesson 1755 — Data Quality and Diversity
- DataFrame
- A DataFrame is essentially a collection of Series (one-dimensional labeled arrays) that all share the same index.
- Lesson 166 — DataFrames: Two-Dimensional Tabular Data Structures
- Datasheets for datasets
- Datasheets for datasets are standardized forms that answer critical questions about a dataset's origins, contents, and intended applications—helping practitioners avoid misuse and understand limitations upfront.
- Lesson 3516 — Introduction to Datasheets for Datasets
- Debug effectively
- Narrow down problems while logs and context are fresh
- Lesson 3064 — Leading vs Lagging Indicators
- Deceptive alignment
- Deceptive alignment is the theoretical risk that a model might learn to behave correctly during training not because it genuinely shares our goals, but because it instrumentally recognizes that appearing aligned helps it get deployed—where it can then pursue different objectives.
- Lesson 3431 — The Scalable Oversight ProblemLesson 3432 — Deceptive Alignment Risk
- decision boundary
- In logistic regression, a decision boundary is the invisible line (or surface) that divides your feature space into regions where the model predicts different classes.
- Lesson 248 — Decision Boundaries in Logistic Regression
- decision stump
- A decision stump is a decision tree with only one split—essentially a tree with just a root and two leaves.
- Lesson 310 — Implementing AdaBoost with Decision Stumps
- decision tree
- A decision tree is a flowchart-like model that makes predictions by asking a series of questions about your data.
- Lesson 285 — Decision Tree Fundamentals and Intuition
- Decoupling
- Separate environment interaction from learning. You might collect 4 steps, then perform 1 gradient update—this ratio is a hyperparameter.
- Lesson 2245 — Training Loop Structure
- deep component
- The deep component is a multi-layer neural network (like the NCF architectures you've learned) that learns low-dimensional embeddings and discovers hidden patterns through non-linear transformations.
- Lesson 2367 — Wide & Deep Networks for Recommendations
- Deeper networks suffer more
- The compounding effect across many layers amplifies the problem
- Lesson 751 — Why Normalization Matters in Deep Networks
- Defense strategies
- Some defenses work better against one type than the other, so understanding the threat model is crucial.
- Lesson 3379 — Targeted vs Untargeted Attacks
- Delimiters
- Delimiters are special characters or sequences that act as visual "fences" to separate prompt components.
- Lesson 1845 — Delimiters and Formatting Markers
- Demographic and protected attribute slices
- Demographic and protected attribute slices are subdivisions of your data based on sensitive characteristics like race, gender, age, disability status, or socioeconomic indicators.
- Lesson 3130 — Demographic and Protected Attribute Slices
- dendrogram
- A dendrogram is a tree-like diagram that shows how hierarchical clustering merges data points or clusters at each step.
- Lesson 356 — Dendrograms and Tree Representations
- dense layer
- In a fully connected layer (also called a dense layer), every single neuron connects to every single neuron in the previous layer.
- Lesson 597 — Fully Connected Layers: Dense Connections
- Dense prediction tasks
- Features at multiple resolutions are perfect for segmentation, detection, and other pixel-level tasks
- Lesson 1354 — Swin Transformer: Hierarchical Architecture
- Deployment Registry
- A central system (like MLflow Model Registry or custom database) that records:
- Lesson 3093 — Model Version Management
- Depth
- Depth refers to the number of transformer encoder blocks stacked together:
- Lesson 596 — Network Architecture Terminology: Depth and WidthLesson 600 — Depth vs Width: Architectural Trade-offsLesson 1349 — ViT Model Variants
- Diagnose weaknesses
- Maybe your model is helpful but often inaccurate
- Lesson 3167 — Multi-Aspect Evaluation with LLM Judges
- diagonal
- Think of it like buying furniture: full covariance is custom-made (expensive, perfect fit), diagonal is adjustable (width/height independent), spherical is one-size (cheap, fast), and tied means all rooms get identical furniture (consistent but less flexible).
- Lesson 371 — Covariance Structure Constraints
- Diagonal Covariance
- The simplest approach treats each action dimension independently. Your network outputs one variance per dimension. This means a 3D action space needs 3 variance values. It's computationally cheap but assumes actions don't correlate.
- Lesson 2316 — Policy Representation for Continuous Actions
- Diagonal patterns
- The model focuses on nearby words—common in language where context is local (e.g., adjectives near nouns).
- Lesson 1059 — Understanding Attention Weight Visualization
- Differencing
- Differencing means subtracting the previous observation from the current one:
- Lesson 2401 — Differencing and Integration
- Diffusion models
- Diffusion models are like an artist who starts with a blurry sketch and refines it with hundreds of careful brush strokes—slow, but the final result is often more detailed and realistic
- Lesson 1537 — Trade-offs: Sample Quality vs Generation Speed
- discount factor γ
- The discount factor γ (gamma) is a number between 0 and 1 that controls this trade-off.
- Lesson 2138 — Discount Factor Gamma
- Discrete reconstruction targets
- The model reconstructs patch-level representations, not raw pixels (which are noisy and high-dimensional)
- Lesson 2573 — Vision Transformer as Reconstruction Target
- discretization
- Binning (also called discretization) transforms continuous variables into discrete categories by dividing their range into intervals or "bins." This simple technique helps machine learning models in several ways:
- Lesson 441 — Binning and Discretization Techniques
- discriminative fine-tuning
- Layer-wise learning rate decay (also called discriminative fine-tuning) applies progressively smaller learning rates to earlier layers and larger rates to later, task-specific layers.
- Lesson 938 — Learning Rate Considerations for Fine-TuningLesson 1177 — Learning Rate and Layer-Wise Decay
- Discriminator
- Receives both the image and the condition, learning to judge whether the image is real AND matches the given condition
- Lesson 1511 — Conditional GANs (cGAN)
- disentanglement
- This separation of factors is called disentanglement.
- Lesson 1452 — β-VAE for Disentanglement
- distributional hypothesis
- The distributional hypothesis is elegantly simple: you shall know a word by the company it keeps.
- Lesson 1118 — The Distributional Hypothesis
- distributional shift
- This is called distributional shift, and it poses a critical challenge to alignment robustness.
- Lesson 3434 — Distributional Shift and Alignment Robustness
- Diverse representation
- Your training data must reflect the populations who will use your system. This means intentionally including data from people with disabilities, varied ages, ethnicities, geographies, and socioeconomic backgrounds.
- Lesson 3494 — Inclusive Design and Accessibility
- Domain adaptation
- Domain adaptation means reusing knowledge from one domain (source) to perform well in another (target), even with limited or no target labels.
- Lesson 1295 — Domain Adaptation and Zero-Shot NERLesson 1979 — Domain Adaptation for Embedding Models
- domain expert persona
- A domain expert persona is a system prompt that positions the model as a specialist in a particular field—like a cardiologist, tax accountant, or software architect.
- Lesson 1857 — Domain Expert Personas
- Domain mismatch
- A model might excel at code but struggle with truthfulness—the average obscures this.
- Lesson 3160 — Leaderboards and Aggregate Scores
- Domain-specific perplexity evaluation
- Domain-specific perplexity evaluation means computing perplexity separately on curated datasets from your target domain, rather than mixing all test data together.
- Lesson 3143 — Domain-Specific Perplexity Evaluation
- Domain-specific rerankers
- Domain-specific rerankers are fine-tuned for particular verticals—medical literature, legal documents, scientific papers, or customer support tickets.
- Lesson 2008 — Reranking Model Selection
- DQN loss function
- The DQN loss function is designed to minimize the TD error across batches of experiences, effectively teaching the network to satisfy the Bellman optimality equation.
- Lesson 2212 — DQN Loss Function Derivation
- dropout2D
- Spatial dropout (also called dropout2D or channel dropout) takes a different approach: instead of randomly zeroing individual values within a feature map, it drops entire feature maps (channels) at once.
- Lesson 746 — Spatial Dropout for Convolutional Layers
- Dual use
- Dual use refers to the reality that AI and machine learning technologies inherently possess the capacity to serve both beneficial and harmful purposes.
- Lesson 3457 — What is Dual Use in AI and Machine Learning?
- Dynamic quantization
- Dynamic quantization is like a chef who preps ingredients (weights) in advance but adjusts seasoning (activation quantization) to taste while cooking each dish.
- Lesson 2632 — Dynamic vs Static Quantization
- Dynamic replanning
- Dynamic replanning means the agent monitors execution in real-time, detects deviations from expected outcomes, and regenerates a new plan on the fly.
- Lesson 2090 — Dynamic Replanning and Error Recovery
E
- Each edge
- Each edge represents a dependency (which values feed into which operations)
- Lesson 643 — The Chain Rule in Computational Graphs
- Each node
- Each node represents a value (variable or operation result)
- Lesson 643 — The Chain Rule in Computational Graphs
- Early layers
- Early layers (close to input) are often more sensitive because they extract low-level features where precision matters
- Lesson 2628 — Where to Apply Quantization in a Model
- Early stopping
- Early stopping is your safety mechanism—it monitors how well your model performs on a validation set during training and stops adding trees when performance stops improving.
- Lesson 319 — Early Stopping and Monitoring in Boosting
- edge list
- An edge list is simply a list of all edges as pairs:
- Lesson 2485 — Graph Representations: Adjacency List and Edge List
- Edges
- Edges are directed arrows showing which task must complete before another can start
- Lesson 2861 — Directed Acyclic Graphs (DAGs)
- effective batch size
- The effective batch size is the total amount of data processed before gradients are averaged and weights are updated — it's the sum of all workers' local batch sizes.
- Lesson 2709 — Effective Batch Size in Data ParallelismLesson 2728 — DDP Debugging and Common PitfallsLesson 2783 — Effective Batch Size vs Physical Batch SizeLesson 2785 — Learning Rate Scaling with Gradient Accumulation
- effective receptive field
- The effective receptive field is the actual region where gradients are significant enough to matter during training.
- Lesson 885 — Effective vs Theoretical Receptive Fields
- Efficiency
- Swin is typically more memory-efficient because attention is strictly local (within windows). PVT computes global attention but reduces the spatial dimension of keys/values, making it moderately more expensive.
- Lesson 1359 — Comparing Hierarchical ViT Architectures
- Empirical Bayes
- Empirical Bayes is the approach where we treat these hyperparameters as tunable parameters rather than choosing them subjectively or using full hierarchical Bayes (which would put priors on the hyperparameters too).
- Lesson 564 — Hyperparameters and Evidence Approximation
- Encoder
- Takes input data and compresses it into a smaller representation (called the "latent code" or "bottleneck")
- Lesson 1429 — What Autoencoders Are and Why They MatterLesson 2574 — MAE: Masked Autoencoder Architecture
- Encoder uses bidirectional attention
- Each token can attend to all other tokens in the input sequence, both before and after its position. This is self-attention without any masking.
- Lesson 1104 — Bidirectional vs Causal Attention
- ENN
- ENN is most conservative, removing only suspicious samples but may not balance classes fully.
- Lesson 542 — Resampling: Undersampling the Majority Class
- epoch
- An epoch is one complete pass through your training dataset.
- Lesson 1708 — Training Duration and Convergence
- Equal Error Rate
- The Equal Error Rate is the point where the false acceptance rate equals the false rejection rate.
- Lesson 2482 — Evaluation Metrics for Speaker Tasks
- Equalized odds
- Equalized odds is more demanding: it requires both equal true positive rates and equal false positive rates across groups simultaneously.
- Lesson 3279 — What is Fairness in Machine Learning?Lesson 3284 — Equalized OddsLesson 3297 — Equal Opportunity and Equalized Odds
- error
- Your error is how far the ball lands from the basket.
- Lesson 120 — ML is Optimization, Not Magic
- Error analysis by subgroup
- Error analysis by subgroup means examining which types of mistakes your model makes for which groups.
- Lesson 3322 — Error Analysis by Subgroup
- Error attribution
- Error attribution is the detective work: identifying which specific decision or component caused the breakdown.
- Lesson 2128 — Trajectory Analysis and Error Attribution
- Error correction opportunity
- If the model makes a small mistake at step 800, it has 200+ more steps to notice and correct it. This self-correcting nature means errors don't compound catastrophically.
- Lesson 1536 — Why Diffusion Models Generate High Quality
- Error handling
- An invalid action (e.g., "open locked door" without a key) requires replanning
- Lesson 1905 — ReAct for Interactive Environments
- Euclidean distance
- Euclidean distance is the default in K-Means — it's the straight-line distance you'd measure with a ruler:
- Lesson 344 — Distance Metrics in K-Means
- Evaluation mode
- Evaluation mode means setting epsilon=0, so your agent always takes the action it believes is best (the greedy action with highest Q-value).
- Lesson 2248 — Evaluation and Testing Protocol
- event
- An event is a set of outcomes—a collection of one or more results we're interested in.
- Lesson 53 — Sample Spaces and Events
- evidence
- Instead of guessing these values, we can learn them from data by maximizing the marginal likelihood (also called evidence).
- Lesson 564 — Hyperparameters and Evidence Approximation
- Exact inference
- Exact inference means computing probabilities of interest without approximation, using two key operations:
- Lesson 579 — Exact Inference: Marginalization and Conditioning
- Exact match
- The predicted entity boundaries and type must match perfectly.
- Lesson 1294 — NER Evaluation Metrics
- Example thinking
- If your SLA is 100ms and inference takes 40ms, your maximum safe timeout is ~50ms (leaving margin for networking and postprocessing).
- Lesson 2917 — Batch Size Selection and Timeout Configuration
- Execution
- Execution is the "worker" that actually performs the computational work—running your preprocessing script on a Spark cluster, training your model on GPUs, or evaluating metrics.
- Lesson 2860 — Pipeline Orchestration vs Execution
- Execution logic
- The actual CUDA kernel performing your operation
- Lesson 2967 — Custom Plugins and Operators
- Expectation violations
- The observation doesn't match what the plan predicted (e.g., expected a customer ID, got "not found")
- Lesson 2090 — Dynamic Replanning and Error Recovery
- Expected Accuracy
- The accuracy a random classifier would achieve given the class distributions
- Lesson 464 — Cohen's Kappa: Agreement Beyond Chance
- Experiment tracking
- Experiment tracking means recording everything needed to reproduce and compare your ML experiments:
- Lesson 148 — Model Versioning and Experiment Tracking Basics
- Expert capacity
- Expert capacity is a hard limit on how many tokens a single expert can process in one forward pass.
- Lesson 1694 — Expert Capacity and Token Dropping
- exploding gradient problem
- While the vanishing gradient problem causes gradients to shrink toward zero, the exploding gradient problem is the opposite nightmare: gradients grow exponentially larger as they backpropagate through layers.
- Lesson 676 — The Exploding Gradient Problem
- Exploitation
- Exploitation means choosing actions that maximize immediate reward based on current knowledge—sticking with the best-known Q-values in Q-Learning or SARSA.
- Lesson 510 — Bayesian Optimization FundamentalsLesson 2185 — The Exploration-Exploitation Dilemma
- Exploration
- Exploration means trying actions you've taken less often, or never, to potentially discover superior strategies.
- Lesson 2185 — The Exploration-Exploitation Dilemma
- exposure bias
- This mismatch between training and testing is called exposure bias.
- Lesson 1029 — Teacher Forcing in Training
- External validators
- External validators are independent mechanisms—like code validators, rule engines, databases, or even other AI models—that check whether an LLM's output meets specific quality criteria before accepting it or triggering another refinement round.
- Lesson 1943 — External Validators in Refinement Loops
F
- F-Beta score
- The F-Beta score is a generalization of the F1 score that lets you control this trade-off using a parameter called beta (β).
- Lesson 457 — F-Beta Score: Weighted Precision-Recall Trade-off
- Failure isolation
- Failure isolation is valuable (one agent failing doesn't crash the system)
- Lesson 2111 — Multi-Agent Systems: Motivation and Use Cases
- Failure to progress
- The diagonal pattern breaks down, causing garbled speech
- Lesson 2467 — Attention Mechanisms in TTS
- Fairness
- Systems should treat all individuals and groups equitably, avoiding discrimination and bias. This means testing across demographics, monitoring disparate impact, and actively mitigating discovered biases.
- Lesson 3487 — Principles of Responsible AI Development
- fairness audit
- A fairness audit is a structured, methodical examination of a machine learning system to identify, measure, and document potential biases across protected groups and different use cases.
- Lesson 3317 — What is a Fairness Audit?
- Faithful Chain-of-Thought
- Faithful Chain-of-Thought means the reasoning trace is not just plausible—it's actually correct at each step.
- Lesson 1872 — Faithful Chain-of-Thought
- fake quantization nodes
- In QAT, your training loop looks similar to standard training but with a crucial addition: fake quantization nodes are actively participating in both forward and backward passes.
- Lesson 2646 — QAT Training Loop Mechanics
- False alarm speech
- detecting speech where there is none
- Lesson 2482 — Evaluation Metrics for Speaker Tasks
- Fast comparison
- Comparing two dataset versions is just comparing hashes (milliseconds vs. hours)
- Lesson 2839 — Content-Addressable Storage for Data
- FastAPI
- FastAPI is particularly popular for ML serving because it includes automatic API documentation, type validation with Pydantic, and async support for handling multiple concurrent requests efficiently.
- Lesson 2894 — REST APIs for Model Serving
- Fault tolerance
- Fault tolerance means your system detects and recovers from failures automatically.
- Lesson 3011 — Fault Tolerance and Graceful Degradation
- Feast
- Feast (Feature Store) is the most popular open-source option.
- Lesson 2890 — Feature Store Tools: Feast, Tecton, and Alternatives
- Feature drift
- Feature drift refers to changes in individual feature distributions—for example, your user_age feature's mean shifts from 35 to 42 over six months.
- Lesson 3028 — Feature Drift vs Covariate Shift
- Feature extraction
- Feature extraction means you freeze BERT's weights completely.
- Lesson 1173 — Fine-Tuning vs Feature Extraction
- feature registry
- The feature registry is the metadata brain.
- Lesson 2883 — Feature Store Architecture Overview
- Feature selection
- The network automatically identifies which connections matter
- Lesson 736 — L1 Regularization for Sparsity
- feature store
- A feature store is a centralized repository that manages, stores, and serves ML features.
- Lesson 2881 — What is a Feature Store and Why It Matters
- features
- Your job as an ML practitioner is to translate real-world data into meaningful numerical representations called features.
- Lesson 117 — The Role of Features and RepresentationsLesson 3266 — Circuits vs Features in Neural Networks
- Feedback
- Feedback is how observations influence the agent's next decision in the ReAct loop.
- Lesson 2063 — Observation Parsing and Feedback
- Feedback mechanisms and recourse
- Feedback mechanisms and recourse are the essential safety valves that let affected individuals interact with AI systems after deployment—reporting problems, appealing unfair outcomes, and requesting explanations.
- Lesson 3495 — Feedback Mechanisms and Recourse
- Few-shot NER
- Few-shot NER means teaching a model to recognize entities with just a handful of labeled examples.
- Lesson 1296 — Few-Shot NER and Prompting Strategies
- Few-shot QA
- Few-shot QA means showing the model 1-3 example question-answer pairs first, then asking your real question.
- Lesson 1310 — QA with Large Language Models
- Final classification layers
- Final classification layers are sensitive because small changes in logits can flip predictions
- Lesson 2628 — Where to Apply Quantization in a Model
- final representation of the CLS token
- After passing through all Transformer layers, the final representation of the CLS token is used as the image representation
- Lesson 1341 — Class Token (CLS Token)
- Find representation gaps
- Discover if certain demographics are underrepresented in your data
- Lesson 3130 — Demographic and Protected Attribute Slices
- Fine-tuned extraction
- Fine-tuned extraction means you continue training CLIP (or just parts of it) on your specific task data.
- Lesson 1401 — Using CLIP as a Feature Extractor
- Fine-tuning
- Fine-tuning means taking the pretrained encoder, attaching a new task-specific head (like a classification layer), and continuing to train the entire model (or parts of it) on your labeled dataset.
- Lesson 1173 — Fine-Tuning vs Feature ExtractionLesson 1953 — RAG vs Fine-Tuning: When to Use EachLesson 2581 — Transfer Learning from Masked Models
- fire module
- The fire module is SqueezeNet's core innovation, consisting of two layers:
- Lesson 924 — SqueezeNet: Fire Modules and Compression
- First and Last Layers
- The input embedding and final classification layers often need higher precision to preserve accuracy
- Lesson 2641 — Quantization of Specific Layer Types
- Fixed maximum sequence length
- This is the critical constraint. When you train with sequences up to length 512, the model only learns positional embeddings for positions 0-511. At inference time, you cannot process a sequence of length 600—there's no embedding for position 550!
- Lesson 1086 — Absolute Positional Embeddings: Advantages and Limitations
- Flexibility
- PVT's spatial reduction ratio is adjustable per layer, offering fine-grained control. Swin's window size is more rigid but benefits from the elegant shifted-window mechanism.
- Lesson 747 — DropConnect and Weight DroppingLesson 1359 — Comparing Hierarchical ViT Architectures
- Flows
- Flows are the top-level containers—think of them as your entire workflow.
- Lesson 2875 — Prefect Architecture and Task API
- force plot
- A force plot is a visual representation that shows how each feature's SHAP value "pushes" the prediction away from the base value (the average model output across your dataset).
- Lesson 3214 — SHAP Force Plots for Individual Predictions
- Formal disclosure programs
- Formal disclosure programs are structured processes where companies invite security researchers to report vulnerabilities confidentially.
- Lesson 3524 — Disclosure Channels and Bug Bounty Programs
- Forward fill
- Forward fill is more common because it mimics how information flows in time: you only know the past, not the future.
- Lesson 433 — Forward Fill and Backward Fill for Time Series
- freezing
- In neural networks, freezing means preventing certain layers from updating their weights during training.
- Lesson 814 — Model Surgery: Freezing Layers and Transfer Learning Setup
- Frozen extraction
- Frozen extraction means you keep CLIP's weights unchanged and simply pass your data through it to get embeddings.
- Lesson 1401 — Using CLIP as a Feature Extractor
- Full fine-tuning
- Full fine-tuning means updating every single parameter in a pretrained LLM during training on your specific task or dataset.
- Lesson 1701 — What Full Fine-Tuning Means for LLMsLesson 1743 — Comparing PEFT Methods: Parameter Count and Performance
- function
- A function is a rule that takes an input and produces exactly one output.
- Lesson 29 — Functions and Continuity
- Function calling
- Function calling is more like providing a menu of specific actions with predefined parameters.
- Lesson 1922 — Function Calling vs JSON Mode
- Function name
- A string identifying which tool to use (e.g., "get_weather")
- Lesson 1923 — Function Schema DefinitionLesson 1925 — Parsing Function Call Responses
G
- GANs
- GANs are like a skilled artist who creates the entire painting in one confident stroke—fast, but sometimes the details aren't perfect
- Lesson 1482 — GANs vs Other Generative ModelsLesson 1537 — Trade-offs: Sample Quality vs Generation Speed
- GAT
- φ computes attention scores, ⊕ is attention-weighted sum, γ applies final transformation
- Lesson 2512 — Message Passing Neural Networks Framework
- Gates
- Gates are learnable on/off switches that control information flow.
- Lesson 1012 — Gates as a Solution to Gradient Flow
- gating network
- The gating network is a lightweight learned module—typically just a simple linear layer followed by a softmax—that takes a token's representation as input and outputs a probability distribution over all available experts.
- Lesson 1690 — Routing Mechanisms in MoE
- Gaussian-Gaussian conjugacy
- With a Gaussian prior on the mean and Gaussian likelihood, the posterior mean is also Gaussian
- Lesson 580 — Conjugate Priors and Analytical Posteriors
- GCN
- φ is identity with normalization, ⊕ is normalized sum, γ applies weights and activation
- Lesson 2512 — Message Passing Neural Networks Framework
- Generate heuristics
- Output node/edge probabilities indicating which choices are promising
- Lesson 2531 — Combinatorial Optimization with GNNs
- Generation Quality
- The LLM receives only the top-K retrieved chunks as context. If those chunks are poorly sized, the generator either gets irrelevant noise (large chunks) or fragmented information (tiny chunks). Your LLM might hallucinate to fill gaps or produce vague answers because the context window contains incomplete information.
- Lesson 1983 — Why Chunking Matters in RAG
- generator
- In the GAN minimax game you just learned, the generator is the creative player.
- Lesson 1471 — Generator Architecture and Role
- Generator loss increasing monotonically
- The discriminator is winning too easily
- Lesson 1502 — Measuring Training Stability
- Global behavior
- Global behavior is extremely non-linear and high-dimensional
- Lesson 3220 — The Local Fidelity Principle
- Global Minimum
- The absolute lowest point across the entire function—the deepest valley in the entire landscape.
- Lesson 95 — Local vs Global Optima
- global norm clipping
- Gradient clipping by norm (also called global norm clipping) works differently than clipping by value.
- Lesson 728 — Gradient Clipping by Norm
- goal
- Your goal is reaching your destination with enough fuel.
- Lesson 2083 — Planning in AI Agents: Problem Formulation
- golden dataset
- A golden dataset is your evaluation bedrock—the unchanging yardstick against which you measure model performance across versions, months, or even years.
- Lesson 3117 — What Makes a Dataset Golden
- Google Cloud Vertex AI Model Registry
- AWS SageMaker Model Registry and Google Cloud Vertex AI Model Registry are fully managed services that integrate seamlessly with their respective cloud ecosystems.
- Lesson 2836 — Alternative Model Registry Solutions
- Graceful degradation
- Graceful degradation means you serve requests at reduced capacity rather than failing completely.
- Lesson 3011 — Fault Tolerance and Graceful Degradation
- gradient
- The gradient is a vector of partial derivatives that tells us how our cost function changes in every direction at once.
- Lesson 211 — The Gradient: Direction of Steepest Ascent
- Gradient alone
- Shows where the stage is sensitive to light changes
- Lesson 3236 — Gradient × Input Method
- gradient checkpointing
- Activation checkpointing (also called gradient checkpointing) solves this by making a deliberate trade-off: store less, compute more.
- Lesson 2786 — Activation Checkpointing Fundamentals
- Gradient flow
- The skip connection provides a direct path for gradients during backpropagation, alleviating the vanishing gradient problem
- Lesson 903 — Residual Learning Formulation
- gradient masking
- This approach is called gradient masking or gradient obfuscation.
- Lesson 3411 — Gradient Masking and Obfuscation
- gradient norm
- The gradient norm is a single number that captures the overall magnitude of all gradients in your network.
- Lesson 726 — Gradient Norm and When to Clip
- Gradients
- Gradients are computed on GPU, then optionally offloaded to CPU
- Lesson 2737 — CPU Offloading in FSDP
- graph
- A graph is a mathematical structure that represents relationships between objects.
- Lesson 2483 — What Is a Graph? Nodes, Edges, and Basic Terminology
- graph Laplacian
- The graph Laplacian is the key matrix here, and its spectral properties (its eigendecomposition) reveal fundamental structural information about the graph.
- Lesson 2493 — Graph Signal Processing and LaplaciansLesson 2498 — Spectral Graph Theory Basics
- GraphSAGE
- φ is identity, ⊕ can be mean/max/LSTM, γ concatenates and transforms
- Lesson 2512 — Message Passing Neural Networks Framework
- greedy action selection
- In reinforcement learning, greedy action selection means always picking the action with the highest estimated value according to your current Q-function or value estimates.
- Lesson 2186 — Greedy Action Selection and Its Limitations
- Grid-based representation
- Every spatial location is represented, not just detected objects
- Lesson 1386 — Vision Transformers in Vision-Language Models
- Ground Truth Labels
- The actual correct outcomes. For loan approval audits, this means knowing who would have repaid; for hiring, who would have succeeded in the role. Ground truth can be tricky when historical data itself reflects bias.
- Lesson 3319 — Data Collection for Audits
- Grounding
- Grounding means constraining the planning process to only reference real, available tools from your tool registry.
- Lesson 2094 — Grounding Plans in Available Tools
- Group fairness
- Group fairness is easier to measure and enforce—you can check approval rates, error rates, or prediction distributions across groups.
- Lesson 3281 — Group Fairness vs Individual Fairness
- Grouped-Query Attention
- Grouped-Query Attention is the middle ground: divide query heads into groups, where each group shares one K/V head.
- Lesson 1610 — Multi-Query and Grouped-Query Attention
- groups
- Think of W&B's organizational system like a filing cabinet: projects are drawers, groups are folders within drawers, tags are colored sticky notes, and notes are detailed labels.
- Lesson 2816 — W&B Run Management and Organization
- Guardrail metrics
- Guardrail metrics are protective measurements that ensure your deployment doesn't cause collateral damage, even if your target metrics improve.
- Lesson 3063 — Guardrail Metrics in Production
H
- hallucination
- Even within their training data, LLMs sometimes generate plausible-sounding but completely fabricated information—a phenomenon called hallucination.
- Lesson 1945 — What RAG Solves: Knowledge Cutoff and Hallucination
- Hamming Loss
- The fraction of labels incorrectly predicted (false positives + false negatives divided by total labels). Lower is better—it captures partial mistakes.
- Lesson 554 — Multi-Label Evaluation Metrics
- Hard negative mining
- Hard negative mining is the practice of deliberately selecting negative samples that are similar to your positive examples, rather than picking them randomly.
- Lesson 2545 — Hard Negative Mining
- Hard negatives
- Hard negatives are examples that are close to your anchor in embedding space but should be pushed away.
- Lesson 1976 — Hard Negatives in Retrieval TrainingLesson 2599 — Hard Negative Mining
- Hardware
- Multi-GPU setups are often essential for models beyond a few billion parameters
- Lesson 1701 — What Full Fine-Tuning Means for LLMs
- Healthcare
- Patient trials are limited and high-stakes
- Lesson 2336 — When to Use Model-Based RL: Sample Efficiency Trade-offs
- Hermes
- Models like Functionary and Hermes are specifically fine-tuned for function calling and work well locally.
- Lesson 1929 — Function Calling with Local Models
- Hidden biases
- The model might reach correct answers through problematic shortcuts
- Lesson 1872 — Faithful Chain-of-Thought
- Hidden layer with linear activations
- Still just straight lines, no matter how many layers you stack (linear combinations of linear functions are still linear)
- Lesson 595 — Why Hidden Layers Matter: Universal Approximation
- High bias
- the model makes strong assumptions by averaging over many points
- Lesson 324 — Choosing K: The Bias-Variance Tradeoff
- High dimensions
- Sometimes optimizing one coordinate at a time is simpler than computing the full gradient
- Lesson 109 — Coordinate Descent
- High-cardinality
- High-cardinality means a categorical variable has many unique values, making standard one-hot encoding impractical.
- Lesson 421 — Handling High-Cardinality Categories
- High-dimensional actions
- Computing max over millions of Q-values is expensive
- Lesson 2249 — From Value Functions to Policies
- High-stakes decisions
- High-stakes decisions are involved (hiring, lending, criminal justice)
- Lesson 3325 — External and Third-Party Audits
- Higher learning rates become safe
- the normalized distribution is more stable
- Lesson 752 — Batch Normalization: Core Concept
- higher-order derivative
- A higher-order derivative is what you get when you differentiate again.
- Lesson 39 — Higher-Order Derivatives
- Historical bias exists in labels
- The ground truth itself may be discriminatory (e.g., past hiring decisions were biased)
- Lesson 3282 — Demographic Parity (Statistical Parity)
- Holm's Method
- A less conservative step-down procedure that adjusts thresholds sequentially based on ranked p-values.
- Lesson 92 — Multiple Testing Correction
- Homogeneous
- Citation network (all nodes are papers, all edges are citations)
- Lesson 2489 — Homogeneous vs Heterogeneous Graphs
- Horizontal patterns
- Consistent direction means monotonic relationship
- Lesson 3213 — SHAP Summary Plots and Feature Importance
- Huber loss
- Huber loss is a hybrid that acts like MSE for small errors (smooth gradients) and like MAE for large errors (outlier protection):
- Lesson 474 — Huber Loss and Robust MetricsLesson 615 — Mean Absolute Error and Huber Loss
- Human Override Mechanisms
- Automated decisions are made but can be contested or overridden by users or operators who see context the model missed.
- Lesson 3491 — Human-in-the-Loop Design Patterns
- Hyperparameters
- Hyperparameters are like the oven temperature and baking time you set beforehand—you control these directly
- Lesson 189 — Parameters vs Hyperparameters
- hyperplane
- In higher dimensions (which we can't visualize), it's called a hyperplane—but the idea stays the same: a flat boundary that cleanly divides your classes.
- Lesson 267 — Linear Separability and Geometric Intuition
I
- Idempotency
- Idempotency means running a task multiple times produces the same result.
- Lesson 2880 — Orchestration Best Practices
- Ignoring directionality
- A significant result in the wrong direction is still a failed experiment.
- Lesson 3078 — Interpreting A/B Test Results
- ImageNet
- ImageNet is a massive image dataset containing over 14 million labeled images spanning 20,000+ categories.
- Lesson 932 — ImageNet and the Data Revolution
- Images
- Images are continuous, high-dimensional arrays of pixels with spatial structure
- Lesson 1374 — Vision-Language Alignment Problem
- Imbalanced data
- Imbalanced data means some classes have many more examples than others.
- Lesson 826 — Handling Imbalanced Data in DataLoaders
- Immediate backfilling
- A new waiting request instantly fills the freed slot in the very next iteration
- Lesson 2983 — Continuous Batching Core Concept
- Immutability
- Immutability is crucial—never modify a published version in place.
- Lesson 3122 — Versioning and Dataset Maintenance
- Imperceptibility
- Changes are typically bounded by a small ε (epsilon) value, making them undetectable to humans
- Lesson 3375 — What Are Adversarial Examples?
- Improved quality
- The discriminator learns richer, class-specific features
- Lesson 1495 — Auxiliary Classifier GAN (AC-GAN)
- In-context learning
- The ability to learn new tasks from examples in the prompt itself
- Lesson 1628 — Emergent Abilities and Phase Transitions
- Inactive states
- Inactive states are temporarily moved to slower CPU memory
- Lesson 1730 — Paged Optimizers for Memory Management
- Inductive bias
- Inductive bias refers to the assumptions a model architecture makes about the data before seeing it.
- Lesson 1345 — Inductive Bias Differences
- inertia
- The metric is called Within-Cluster Sum of Squares (WCSS), also known as inertia.
- Lesson 339 — K-Means Objective Function
- Inference
- Inference is like running that factory every day: each individual prediction uses far less energy than training, but when you serve billions of predictions to millions of users continuously, the cumulative impact can dwarf the training cost.
- Lesson 3471 — Training vs Inference Environmental Costs
- Inference Speedup
- Combining reduced computation with smaller memory footprints means faster predictions. This matters for real-time applications like autonomous driving or video processing.
- Lesson 2666 — Why Prune: Benefits and Trade-offs
- infinity norm
- The infinity norm is simply the largest absolute value among all components.
- Lesson 4 — Vector Norms and Distance Metrics
- InfoNCE loss
- The InfoNCE loss is the mathematical heart of this framework:
- Lesson 2547 — Contrastive Learning Framework and InfoNCE Loss
- Informed consent
- Informed consent means users understand what data you're collecting, why, how it will be used, and what risks exist.
- Lesson 3492 — Consent and Data Practices
- Infrastructure Blocks
- Infrastructure Blocks are reusable configuration templates stored in Prefect Cloud.
- Lesson 2876 — Prefect Cloud and Deployment Patterns
- initialization
- This choice—called initialization—matters far more than you might expect.
- Lesson 340 — Initialization Methods
- injectivity
- This property is called injectivity—different inputs always produce different outputs.
- Lesson 2513 — GIN: Graph Isomorphism Networks
- inner loop
- The inner loop is where your model learns a specific task.
- Lesson 2609 — MAML's Inner and Outer Loop
- Input
- A natural language instruction ("Summarize this article," "Explain quantum physics simply," "Write a poem about spring")
- Lesson 1316 — Fine-Tuning for SummarizationLesson 1749 — What Is Instruction Tuning?
- Input encoding
- Historical values are tokenized with positional encodings that preserve temporal ordering
- Lesson 2424 — TimeGPT Architecture and Pretraining Strategy
- instruction dataset
- An instruction dataset consists of (instruction, response) pairs that teach the model this behavior.
- Lesson 1230 — Instruction Dataset Construction
- Instruction tuning
- Instruction tuning is a fine-tuning phase where you train the pre-trained model on datasets of (instruction, desired response) pairs.
- Lesson 1749 — What Is Instruction Tuning?
- Instruction-tuned model
- Starting from a guided tour route—easier to follow and modify, but some routes are already preferred
- Lesson 1236 — Further Fine-Tuning: Starting from Base or Instruction
- Instruction-tuned models
- Instruction-tuned models (like ChatGPT) are fine-tuned specifically to interpret commands as tasks to execute, not patterns to complete.
- Lesson 1228 — Base Model Behavior: Completion vs Following InstructionsLesson 1233 — When to Use Base vs Instruction-Tuned Models
- INT4 quantization
- INT4 quantization represents each weight using only 4 bits (16 possible values), achieving an 8× compression ratio.
- Lesson 2662 — INT4 and Sub-Byte Quantization
- Integrity verification
- The hash serves as a tamper-proof checksum
- Lesson 2839 — Content-Addressable Storage for Data
- Intent ambiguity
- The same model can classify medical images or power surveillance
- Lesson 3458 — Historical Examples of Dual Use Technology
- Intentionality
- Unlike random noise, adversarial perturbations are specifically optimized to cause misclassification
- Lesson 3375 — What Are Adversarial Examples?
- Interleaved image-text training
- Interleaved image-text training means feeding your model sequences where images and text tokens appear in their natural order, mixed together.
- Lesson 1418 — Interleaved Image-Text Training
- internal covariate shift
- This phenomenon, called internal covariate shift, forces you to use small learning rates and careful weight initialization.
- Lesson 751 — Why Normalization Matters in Deep NetworksLesson 752 — Batch Normalization: Core ConceptLesson 873 — Batch Normalization in CNNs
- Interpretability
- Interpretability refers to the degree to which a human can understand the internal mechanics of a model—how it makes decisions from input to output.
- Lesson 736 — L1 Regularization for SparsityLesson 3183 — What is Model Interpretability?
- intersection
- The intersection is where both circles overlap.
- Lesson 947 — Intersection over Union (IoU)
- intrinsic dimensionality
- In neural networks, many weight matrices have redundant structure — the actual "information content" or intrinsic dimensionality is much lower than the full matrix size suggests.
- Lesson 1712 — Low-Rank Matrix Factorization Intuition
- Invalidation
- Invalidation is critical—stale predictions hurt accuracy.
- Lesson 2919 — Result Caching Strategies
- IQR
- Best when data has outliers or is skewed
- Lesson 77 — Descriptive Statistics: Spread and Variability
- isotropic Gaussian
- An isotropic Gaussian means the distribution is:
- Lesson 1529 — Why the Final Distribution is Gaussian
- It's directly tied to prediction quality
- Lower perplexity means the model assigns higher probability to the actual next tokens
- Lesson 3137 — What Perplexity Measures in Language Models
- Item-based
- The clerk looks at books you've purchased, finds books that were frequently bought by the same groups of people, then recommends those related books.
- Lesson 2350 — User-Based vs Item-Based Approaches
- Iterative DPO
- Iterative DPO means running multiple rounds where you:
- Lesson 1816 — Iterative DPO and Online Alignment
J
- Jacobian matrix
- The Jacobian matrix is a mathematical structure that organizes all these partial derivatives in one place.
- Lesson 635 — Jacobian Matrices in Backpropagation
- jailbreak
- A jailbreak is a carefully crafted input that tricks a large language model into bypassing its built-in safety guardrails—producing harmful, biased, or policy-violating outputs it was explicitly designed to refuse.
- Lesson 3413 — What Are Jailbreaks and Why They Matter
- Joblib
- Joblib is a library designed specifically for efficiently saving and loading Python objects, particularly large NumPy arrays (which is exactly what ML models contain).
- Lesson 186 — Saving and Loading Models with Joblib
- JSON mode
- JSON mode is like giving someone a blank form and saying "fill this out in valid JSON format." The model generates free-form JSON text that you then parse and validate.
- Lesson 1922 — Function Calling vs JSON Mode
- Just right
- The model converges efficiently—fast enough to be practical, stable enough to reliably find a good minimum.
- Lesson 101 — Learning Rate and Step Size
K
- K-means clustering
- K-means clustering is a popular approach: you cluster your weight values into k groups, then use the cluster centroids as your quantization levels.
- Lesson 2624 — Uniform vs Non-Uniform Quantization
- K-shot
- K-shot refers to the number of labeled examples you have per class
- Lesson 2584 — N-Way K-Shot Terminology
- kernel function
- A kernel function is a mathematical shortcut.
- Lesson 279 — The Kernel Function Definition
- Kernel launch reduction
- Each kernel launch has overhead (~5-20 microseconds). Fusing 3 layers means 3× fewer launches.
- Lesson 2959 — Layer and Tensor Fusion
- key
- Think of it like searching a library: your query is "books about transformers," each book's key is its title and topic tags, and the value is the book's actual content.
- Lesson 1051 — Query, Key, Value: The Three Vectors
- Keys and values
- Keys and values are downsampled (e.g., by 2× or 4×) using convolution or pooling
- Lesson 1358 — Pyramid Vision Transformer (PVT)
- Keyword-enriched version
- The chunk with extracted key terms highlighted
- Lesson 1995 — Multi-Representation Chunking
- knowledge graph
- A knowledge graph is a structured representation of facts as entities (nodes) and relations (edges).
- Lesson 2101 — Entity Memory and Knowledge GraphsLesson 2529 — Knowledge Graph Reasoning
- Kubeflow Pipelines SDK
- The kfp Python package lets you author pipeline components, compile pipelines into YAML specifications, and submit them to the Kubeflow Pipelines backend for execution on your Kubernetes cluster.
- Lesson 2877 — Kubeflow Pipelines Overview
- KV cache eviction
- KV cache eviction is the process of selectively removing cached positions when you hit memory limits, keeping only the most valuable information.
- Lesson 1678 — KV Cache Eviction Strategies
L
- L1 norm
- The L1 norm is the sum of the absolute values of all components in a vector.
- Lesson 4 — Vector Norms and Distance Metrics
- L2 Cache
- A 40-80MB buffer sitting between compute cores and VRAM. When the same data is accessed multiple times (like attention weights in transformers), L2 cache can serve it 10-20x faster than VRAM. Cache locality is why compact data layouts matter.
- Lesson 2935 — Understanding GPU Memory Hierarchy for Inference
- L2 norm
- The L2 norm is the square root of the sum of squared components—the "straight-line" distance.
- Lesson 4 — Vector Norms and Distance Metrics
- Label corrections
- A team member fixes 500 mislabeled samples. Your model improves, but last week's experiment is now impossible to recreate.
- Lesson 2837 — Why Data Versioning Matters in ML
- Lagging indicators
- Lagging indicators are the actual business outcomes you care about—revenue, conversion rates, customer retention—but they take days, weeks, or even months to materialize.
- Lesson 3064 — Leading vs Lagging Indicators
- Language matters
- English tolerates lowercasing better than German (where nouns are capitalized)
- Lesson 1269 — Tokenizer Normalization and Preprocessing
- large margin
- A large margin means your boundary has buffer space—it's more likely to generalize well to new, unseen data
- Lesson 268 — The Concept of Margin
- Larger patches
- Larger patches are computationally cheaper but may miss fine-grained patterns.
- Lesson 1347 — Resolution and Patch Size Trade-offs
- Latency SLOs
- Often expressed as percentiles (p50, p95, p99). A p99 < 200ms target means 99% of requests must complete under 200ms.
- Lesson 2932 — Service Level Objectives (SLOs) and Budget Allocation
- Latent imagination
- Latent imagination is the process of planning by "imagining" future trajectories in latent space.
- Lesson 2337 — World Models and Latent Imagination
- Latent interpolation
- Latent interpolation means creating a smooth path between two images in latent space.
- Lesson 1577 — Latent Space Interpolation and Editing
- Layer freezing
- Layer freezing means locking certain layers' weights so they don't update during training, while allowing others to learn from your new data.
- Lesson 937 — Layer Freezing StrategiesLesson 941 — Domain Adaptation Challenges
- Layer-wise attention analysis
- Layer-wise attention analysis means systematically examining how attention weights change across layers, revealing a progression from low-level syntactic patterns to high-level semantic relationships.
- Lesson 3258 — Layer-Wise Attention Analysis
- lazy learning
- This is called lazy learning because the algorithm postpones all computational work until prediction time, unlike eager learning methods (like SVMs or Random Forests) that do heavy computation during training.
- Lesson 321 — The Instance-Based Learning Paradigm
- Leading indicators
- Leading indicators are early warning signals you can measure immediately or soon after deployment—things like prediction latency, confidence scores, input distribution shifts, or user engagement patterns.
- Lesson 3064 — Leading vs Lagging Indicators
- leaf node
- Each leaf node represents a word in your vocabulary
- Lesson 1122 — Hierarchical Softmax for Word2Vec
- Learned representations
- The model discovers its own internal "language" for meaning
- Lesson 1035 — Applications: Machine Translation
- Learning
- Learning is the process of adjusting these parameters (through many attempts) to minimize your misses.
- Lesson 120 — ML is Optimization, Not Magic
- learning curve
- A learning curve is a graph that shows how your model's performance (like accuracy or loss) changes as training progresses—typically measured across epochs or training set size.
- Lesson 519 — What Learning Curves Reveal
- Learning Rate
- Your neural network's step size when updating Q-values. Too high and your agent forgets past knowledge catastrophically; too low and learning crawls. In DQN variants with replay buffers, lower learning rates (e.g., 0.00025 vs 0.001) often work better because samples are reused multiple times.
- Lesson 2235 — Hyperparameter Sensitivity in DQN VariantsLesson 2272 — REINFORCE Convergence Properties
- Least Squares Criterion
- The Least Squares Criterion is simply the principle that the best line is the one that minimizes the sum of squared errors.
- Lesson 192 — The Least Squares Criterion
- length extrapolation
- This is called length extrapolation and solves a major limitation of absolute position encodings.
- Lesson 1655 — Rotary Position Embeddings (RoPE)
- LightGBM
- LightGBM is typically the fastest, especially on large datasets with many rows.
- Lesson 320 — Comparing Boosting Libraries: XGBoost vs LightGBM vs CatBoost
- likelihood
- The likelihood is the probability of all observed targets {y₁, y₂, ..., yₙ} given parameters θ (weights, bias) and inputs X:
- Lesson 329 — Bayes' Theorem and Posterior ProbabilityLesson 559 — Likelihood Function for Regression
- limit
- A limit describes what value a function approaches as its input gets arbitrarily close to some point.
- Lesson 30 — Limits: The Foundation of Derivatives
- Limitation
- LLM requests vary wildly in length. A short prompt might take 100ms while a long generation takes 30 seconds. Round-robin can leave servers idle while others are overloaded.
- Lesson 3006 — Load Balancing Strategies for LLM Services
- Limited data
- Your training set is just a sample, never the complete universe of possibilities. As you learned with sampling distributions, samples contain uncertainty.
- Lesson 122 — ML Models as Approximations
- Limited Expertise
- Many alignment tasks require specialized knowledge (medicine, law, coding). Finding expert annotators for every domain is impractical.
- Lesson 1817 — Limitations of Human Feedback and Motivation for RLAIF
- linear decision boundary
- A linear decision boundary is a straight line (or flat plane in higher dimensions).
- Lesson 238 — Decision Boundaries and Separability
- Linear independence
- Linear independence means vectors provide genuinely different directions—none can be created by combining the others using scalar multiplication and addition.
- Lesson 10 — Linear Independence and Span
- Linear probing
- Linear probing is a diagnostic approach: you freeze the pretrained encoder completely and train only a simple linear classifier on top of the extracted features.
- Lesson 2581 — Transfer Learning from Masked Models
- Linear separability
- Linear separability means you can draw a straight line that perfectly separates all red dots on one side from all blue dots on the other, with no mistakes.
- Lesson 267 — Linear Separability and Geometric Intuition
- Linearly separable data
- Linearly separable data means you can draw a straight line that perfectly separates the classes.
- Lesson 238 — Decision Boundaries and Separability
- linkage
- The linkage criterion ('ward', 'complete', 'average', or 'single')
- Lesson 362 — Implementing Hierarchical Clustering with Scikit-learn
- Local linearity assumption
- Gradients assume your model is locally linear around the input. But neural networks are highly nonlinear. A tiny perturbation might hit a decision boundary, causing gradients to point in misleading directions.
- Lesson 3234 — Why Raw Gradients Are Noisy
- Local Maximum
- A point higher than all nearby points. Picture a hilltop.
- Lesson 95 — Local vs Global Optima
- Local Minimum
- A point where the function value is lower than all nearby points, but not necessarily the lowest overall. Think of it as a valley surrounded by hills.
- Lesson 95 — Local vs Global Optima
- Local Outlier Factor
- The Local Outlier Factor is the workhorse algorithm here.
- Lesson 375 — Density-Based Anomaly Detection
- locking down exact package versions
- In ML, where tiny numerical differences can compound through training, locking down exact package versions is critical for reproducibility.
- Lesson 2851 — Managing Python Dependencies with requirements.txt
- log-odds
- The log-odds (or logit) is just the natural logarithm of the odds:
- Lesson 253 — Probabilistic Interpretation and Odds
- Logical Leaps
- Steps don't follow logically from previous ones. The model might jump from "A is true" to "C is true" without establishing B, creating gaps in the reasoning chain.
- Lesson 1874 — Chain-of-Thought Hallucinations and Errors
- logits
- Given K raw scores (called logits) from your model — one per class — the softmax function does two things:
- Lesson 261 — The Softmax Function Definition
- Long-term alignment
- Long-term alignment means honest critique and pushing through discomfort—better outcomes, but potentially negative immediate feedback.
- Lesson 3445 — Short-Term vs Long-Term Alignment
- Low bias
- the model makes few assumptions and can capture complex patterns
- Lesson 324 — Choosing K: The Bias-Variance Tradeoff
- Low or negative value
- vectors are dissimilar → low relevance
- Lesson 1052 — Computing Attention Scores with Dot Products
- low-fidelity
- These cheaper evaluations are called low-fidelity approximations.
- Lesson 516 — Multi-Fidelity Optimization
- Low-rank approximation
- Low-rank approximation means we keep only the top k singular values and their corresponding columns/rows from U and V^T, then reconstruct an approximate version of the original matrix.
- Lesson 24 — Matrix Approximation with SVD
- Lower temperatures
- Lower temperatures are safer but transfer less nuance.
- Lesson 2692 — Practical Distillation: Hyperparameters and Pitfalls
M
- Main effects
- The standalone contribution of each feature (diagonal elements)
- Lesson 3216 — SHAP Interaction Values
- Majority voting
- The most common approach—each agent casts one vote, and the option with the most votes wins. Simple but assumes all agents are equally reliable.
- Lesson 1880 — Majority Voting ImplementationLesson 2116 — Consensus and Voting Mechanisms
- Manhattan distance
- Sum of absolute differences along each dimension. Useful when features are independent.
- Lesson 2343 — Similarity Metrics for Content Matching
- Many-shot prompting
- Many-shot prompting is like showing several route examples—now the pattern becomes unmistakable.
- Lesson 1838 — One-Shot vs Many-Shot Trade-offs
- margin
- Recall from the previous lesson that the margin is the distance from the decision boundary to the nearest data point.
- Lesson 268 — The Concept of MarginLesson 269 — Hard-Margin SVM Objective
- Marginalization
- Marginalization is like "summing out" or "integrating out" variables you don't care about.
- Lesson 579 — Exact Inference: Marginalization and Conditioning
- Markov property
- The Markov property means that to compute the image at timestep t, you only need the image from timestep t-1 — not the entire history of how we got there.
- Lesson 1525 — The Markov Chain of Noise Addition
- Masked models like BERT
- Masked models like BERT are trained to fill in missing words when they can see context from both directions.
- Lesson 1198 — Why Autoregressive for Generation Tasks
- Match human hearing
- The mel scale aligns with how we perceive pitch and frequency
- Lesson 2464 — Mel Spectrograms as Intermediate Representation
- Materialization
- Materialization is the ongoing process of computing feature values from raw data and writing them to your feature store—both offline (for training) and online (for serving).
- Lesson 2887 — Feature Materialization and Backfilling
- matrix
- A matrix is a rectangular grid of numbers arranged in rows and columns.
- Lesson 1 — Scalars, Vectors, and Matrices: DefinitionsLesson 775 — What is a Tensor?
- Matrix dimensions
- If W is (n_out × n_in), x is (n_in × 1), and dL/dz is (n_out × 1), then dL/dW is correctly (n_out × n_in).
- Lesson 633 — Backpropagation for Fully Connected Layers
- matrix factorization
- Classical collaborative filtering using matrix factorization (like SVD and ALS) represents users and items as low-dimensional vectors whose dot product approximates ratings.
- Lesson 2363 — From Matrix Factorization to Neural Networks
- Matrix Multiplication
- Matrix Multiplication is the heart of ML computations.
- Lesson 158 — Linear Algebra Operations
- Matthews Correlation Coefficient
- You've learned many classification metrics—accuracy, precision, recall, F1 score, Cohen's Kappa—but Matthews Correlation Coefficient is special because it considers all four cells of the confusion matrix equally.
- Lesson 465 — Matthews Correlation Coefficient
- Mean
- the average of all non-missing values
- Lesson 66 — Uniform DistributionLesson 432 — Simple Imputation: Mean, Median, and Mode
- Mean shift
- Your feature that averaged 100 is now averaging 120
- Lesson 3053 — Statistical Summary Monitoring
- Measuring alignment
- Measuring alignment means creating tests and metrics to assess whether a model genuinely pursues intended goals rather than exploiting loopholes or pursuing unintended instrumental goals.
- Lesson 3436 — Measuring and Evaluating Alignment
- Median
- Better when data has outliers or is skewed
- Lesson 76 — Descriptive Statistics: Central TendencyLesson 432 — Simple Imputation: Mean, Median, and Mode
- Median imputation
- Median imputation is better when your data has outliers or is skewed.
- Lesson 432 — Simple Imputation: Mean, Median, and Mode
- Medical diagnosis
- Missing a disease (false negative) in one demographic group more often than another is ethically problematic
- Lesson 3283 — Equal Opportunity
- mel scale
- The mel scale is a perceptually-motivated transformation that maps physical frequencies to how humans actually experience pitch.
- Lesson 2439 — Mel Scale and Mel-Spectrograms
- Memory Reduction
- Storing fewer weights directly reduces model size. A pruned model occupies less RAM and disk space—critical for mobile devices or edge deployment where memory is constrained.
- Lesson 2666 — Why Prune: Benefits and Trade-offs
- Memory savings
- A 512×512 RGB image contains 786,432 values. The autoencoder might compress this to a 64×64×4 latent representation—just 16,384 values. That's 48× fewer dimensions to process at each diffusion step.
- Lesson 1575 — Computational Benefits of Latent Diffusion
- Message passing
- Message passing is the mechanism by which agents send and receive information, while communication protocols define the rules and formats for these exchanges.
- Lesson 2112 — Agent Communication Protocols and Message Passing
- message passing framework
- The message passing framework is the fundamental operation that powers most Graph Neural Networks.
- Lesson 2496 — The Message Passing Framework
- Metadata and lineage tracking
- Metadata and lineage tracking means recording detailed information about what data was used, how it was transformed, which models were trained, and when each step occurred throughout your ML pipeline.
- Lesson 2862 — Metadata and Lineage Tracking
- Metadata enrichment
- Metadata enrichment is the practice of tagging each chunk with extra information about its origin and context—like keeping a library card with each page you tear out of a book.
- Lesson 1993 — Metadata Enrichment
- Method of Moments
- The Method of Moments is a parameter estimation technique that works by setting sample statistics (like the mean or variance you calculate from your data) equal to their theoretical counterparts, then solving for the unknown parameters.
- Lesson 86 — Method of Moments
- min-max scaling
- Min-Max Normalization (also called min-max scaling) squeezes all your feature values into a specific range by finding the minimum and maximum values, then rescaling everything proportionally between them.
- Lesson 408 — Min-Max Normalization
- mini-batch
- Stochastic Gradient Descent (SGD) takes a smarter approach: instead of computing the exact gradient from all data, it estimates the gradient using a small random subset called a mini-batch (often 32, 64, or 256 examples).
- Lesson 105 — Stochastic Gradient Descent Basics
- Mini-batch gradient descent
- Mini-batch gradient descent is the "just right" middle ground—it computes gradients on small batches of training examples.
- Lesson 684 — Mini-Batch Gradient Descent
- miniImageNet
- miniImageNet is the most popular few-shot learning benchmark.
- Lesson 2594 — Few-Shot Learning Benchmarks
- Minimal parameters
- Only the prefix vectors are trainable
- Lesson 1739 — Prefix Tuning: Prepending Learnable Vectors
- Mish activation
- A smoother alternative to ReLU that helps gradients flow
- Lesson 965 — YOLOv4 and YOLOv5: Speed and Accuracy Advances
- Mixed precision quantization
- Mixed precision quantization means applying different quantization bit-widths to different parts of your model based on how sensitive each layer is to reduced precision.
- Lesson 2629 — Mixed Precision Quantization
- Mixout
- Mixout is a dropout-inspired technique that randomly keeps some weights at their pretrained values during fine-tuning.
- Lesson 1183 — Catastrophic Forgetting and Regularization
- ML Development Lifecycle
- The ML Development Lifecycle describes this repeating journey through several connected stages.
- Lesson 135 — The ML Development Lifecycle Overview
- ML pipeline
- An ML pipeline is an automated workflow that orchestrates the entire machine learning lifecycle—from data ingestion and preprocessing, through model training and evaluation, to deployment and monitoring.
- Lesson 2857 — What is an ML Pipeline?
- MMR
- MMR is a classic technique that balances relevance with diversity.
- Lesson 2009 — Diversity in Reranking
- MoCo
- Memory is largely independent of negative sample count; runs efficiently on 8 GPUs
- Lesson 2557 — SimCLR vs MoCo: Comparative Analysis
- mode collapse
- This phenomenon is called mode collapse—the model's output distribution collapses from diverse, creative responses down to a narrow set of high-reward patterns.
- Lesson 3441 — Mode Collapse and Response Diversity
- Mode imputation
- Mode imputation is ideal for categorical variables (like "color" or "city") or discrete counts.
- Lesson 432 — Simple Imputation: Mean, Median, and Mode
- Model artifacts
- The trained model files themselves
- Lesson 148 — Model Versioning and Experiment Tracking Basics
- Model awareness
- The model learns to treat these differently—padding tokens don't contribute to loss, <eos> triggers stopping conditions.
- Lesson 1648 — Handling Special Tokens
- model card
- A model card is a structured document that accompanies a machine learning model, providing critical information about its development, capabilities, limitations, and appropriate use cases.
- Lesson 3511 — Introduction to Model Cards
- model cards
- Introduced by Mitchell et al., model cards are standardized reports describing a trained model.
- Lesson 3490 — Transparency and Documentation Standards
- model complexity curve
- A model complexity curve is a plot that shows how both training error and validation error change as you increase the complexity of your model.
- Lesson 525 — Model Complexity Curves
- Model parallelism
- Model parallelism is tearing a single book into chapters and having readers process them sequentially, passing results along.
- Lesson 2755 — Model Parallelism vs Data Parallelism
- Model Protection
- The ML model itself can be kept confidential from unauthorized parties
- Lesson 3373 — Trusted Execution Environments
- model registry
- A model registry is a centralized repository that stores trained machine learning models along with their metadata, versions, and deployment information.
- Lesson 2828 — Model Registry Fundamentals
- Model serving
- Model serving is the process of deploying trained machine learning models into production environments where they can receive input data and return predictions in real time or in batches.
- Lesson 2891 — What is Model Serving?
- Model versioning
- TensorFlow Serving automatically discovers and loads new model versions from a specified directory. Each version is numbered (e.g., /models/my_model/1, /models/my_model/2). By default, it serves the latest version but can maintain multiple versions simultaneously. Clients can request specific versions or use policies like "serve the latest stable version."
- Lesson 148 — Model Versioning and Experiment Tracking BasicsLesson 2908 — TensorFlow Serving Architecture
- Momentum encoders
- Momentum encoders are a clever solution to keep these stored embeddings consistent.
- Lesson 2541 — Momentum Encoders and Memory Banks
- Monitor training
- Watch for signs one network is dominating (discriminator loss near 0 or 1, generator loss exploding)
- Lesson 1503 — Learning Rate Balance
- More good data beats better algorithms
- A linear model with 10,000 clean, diverse examples often beats a neural network with 500 noisy ones.
- Lesson 121 — The Data-Centric View of ML
- Multi-aspect evaluation
- Multi-aspect evaluation means judging outputs across separate, well-defined dimensions:
- Lesson 3167 — Multi-Aspect Evaluation with LLM Judges
- Multi-class
- Multi-class is like choosing your meal from a restaurant menu—you pick one entrée from several options.
- Lesson 549 — Multi-Label vs Multi-Class: Key Differences
- Multi-label
- Multi-label is like choosing toppings for a pizza—you can select multiple toppings or none at all, and each choice is independent.
- Lesson 549 — Multi-Label vs Multi-Class: Key Differences
- Multi-Model Serving
- A single TensorFlow Serving instance can host multiple different models concurrently. It monitors a model base path and automatically loads all models it discovers. Each model gets its own endpoint (e.g., /v1/models/model_a:predict, /v1/models/model_b:predict).
- Lesson 2908 — TensorFlow Serving Architecture
- Multi-scale inference
- Multi-scale inference means running your trained model on the same image at different resolutions (scales), then combining the results.
- Lesson 985 — Multi-Scale Inference and Test-Time Augmentation
- Multi-tenancy
- Multi-tenancy means multiple "tenants" (clients, teams, or model instances) share the same physical hardware—but each must feel like they have dedicated resources.
- Lesson 3013 — Multi-Tenancy and Isolation in Shared Infrastructure
- Multinomial Naive Bayes
- While Gaussian Naive Bayes works well for continuous features (like height or temperature), Multinomial Naive Bayes is designed specifically for count data—features that represent how many times something occurs.
- Lesson 332 — Multinomial Naive Bayes for Count Data
- Multiple output channels
- Multiple output channels means using multiple different filters (kernels) on the same input, producing multiple feature maps simultaneously.
- Lesson 859 — Multiple Output Channels
N
- N-way
- N-way refers to the number of classes you need to distinguish between
- Lesson 2584 — N-Way K-Shot Terminology
- Named entity recognition
- Context on both sides helps identify if "Jordan" is a person or country
- Lesson 1024 — Bidirectional LSTMs and GRUs
- Nash Equilibrium
- Nash Equilibrium is the theoretical endpoint where neither player can improve by changing their strategy alone—they've reached a stable standoff.
- Lesson 1474 — Nash Equilibrium in GANs
- Natural language inference
- Determining if one sentence contradicts or supports another.
- Lesson 1148 — The [SEP] Token for Segment Separation
- Natural text generation
- The model learns to generate coherent sequences without requiring a separate encoder
- Lesson 1200 — Decoder-Only Design: Why GPT Diverged from BERT
- Nearest Neighbor Baseline
- The Nearest Neighbor Baseline is the most straightforward few-shot learning method.
- Lesson 2590 — Nearest Neighbor Baseline
- Negative determinant
- The transformation flips orientation (like mirroring)
- Lesson 14 — Determinants and Their Properties
- Neighborhood aggregation
- Neighborhood aggregation is the fundamental mechanism that lets a node learn from its local graph structure by gathering information from the nodes it's connected to.
- Lesson 2492 — Neighborhood Aggregation IntuitionLesson 2495 — Graph Structure and Neighborhood Aggregation
- Network architecture sensitivity
- The gradient signal must backpropagate through many layers. Each layer's weights, biases, and activation choices distort this signal. Deep networks especially suffer—by the time gradients reach the input, they've been transformed through dozens of nonlinear operations, accumulating numerical artifacts.
- Lesson 3234 — Why Raw Gradients Are Noisy
- No built-in locality bias
- Transformers don't assume nearby patches are related; they learn relationships from data
- Lesson 1337 — From CNNs to Vision Transformers
- No generalization to longer sequences
- Unlike relative positional encoding schemes (not yet covered), absolute embeddings don't inherently understand the relationship between positions. Position 10 and position 11 are encoded as completely independent vectors.
- Lesson 1086 — Absolute Positional Embeddings: Advantages and Limitations
- No preference learning
- It doesn't know whether response A is better than response B for the same query
- Lesson 1760 — From Instruction Tuning to Alignment
- No text generation
- The model never creates new words or paraphrases
- Lesson 1298 — Extractive QA Fundamentals
- node
- Each node is an independent server with its own:
- Lesson 642 — Forward Pass Through a Computational GraphLesson 2791 — Multi-Node Training Architecture
- Nodes
- Nodes are individual tasks (data loading, preprocessing, training, evaluation, deployment)
- Lesson 2861 — Directed Acyclic Graphs (DAGs)
- Noise points
- Noise points are labeled -1—these are outliers in regions too sparse to form clusters.
- Lesson 354 — Implementing and Evaluating Density-Based Clustering
- Nominal categories
- Nominal categories are just names or labels with no intrinsic order.
- Lesson 418 — Ordinal vs Nominal Categories
- Non-linearity
- If residuals form a curve, your linear model is trying to fit a curved relationship
- Lesson 477 — Residual Analysis and Diagnostic Plots
- non-linearly separable
- This is called a non-linearly separable problem.
- Lesson 592 — Perceptron Limitations: The XOR Problem
- Non-separable data
- Non-separable data means no straight line works — the classes overlap or interweave.
- Lesson 238 — Decision Boundaries and Separability
- nonlinear decision boundary
- A nonlinear decision boundary is curved or wiggly.
- Lesson 238 — Decision Boundaries and Separability
- norm
- A norm is a function that assigns a non-negative length or magnitude to a vector.
- Lesson 4 — Vector Norms and Distance Metrics
- Normalization
- Normalization is the process of converting text into a canonical form so your tokenizer treats semantically identical inputs consistently.
- Lesson 1650 — Normalizing Input Text
- Not depending on batch size
- statistics are computed per sample, per group
- Lesson 759 — Group Normalization
- notes
- Think of W&B's organizational system like a filing cabinet: projects are drawers, groups are folders within drawers, tags are colored sticky notes, and notes are detailed labels.
- Lesson 2816 — W&B Run Management and Organization
- Numbers and special characters
- Numbers and special characters are particularly inefficient — long numbers might tokenize as individual digits, wasting precious context slots.
- Lesson 1651 — Tokenization and Context Window
- Numerical stability
- Direct matrix inversion can be unstable when XᵀX is nearly singular (non-invertible). A more robust alternative uses np.linalg.lstsq(), which employs QR decomposition or SVD under the hood:
- Lesson 202 — Computing the Normal Equation in NumPy
O
- object queries
- In the Detection Transformer (DETR), object queries are learnable embeddings that act as "slots" or "question marks" for potential objects in an image.
- Lesson 1366 — Object Queries and Learned Positional Embeddings
- Object structure
- If you see part of a wheel, the rest is probably circular
- Lesson 2571 — Masked Image Modeling: Core Concept
- objective function
- The objective function is K-Means' scoring system—it measures how tightly grouped points are within their assigned clusters.
- Lesson 339 — K-Means Objective Function
- Observability
- Observability means instrumenting your system so you can understand why it's behaving a certain way, not just that something is wrong.
- Lesson 2868 — Pipeline Monitoring and ObservabilityLesson 3014 — Monitoring and Observability at Scale
- Observations
- Observations are acoustic features (MFCCs or log-mel spectrograms extracted from audio frames)
- Lesson 2449 — Hidden Markov Models for ASR
- Offline evaluation
- Offline evaluation is fast, cheap, and reproducible—perfect for rapid iteration and comparing dozens of model variants
- Lesson 2383 — Offline vs Online Evaluation Trade-offs
- Offline metrics
- Offline metrics are computed on static, historical datasets—typically your validation or test sets.
- Lesson 3059 — What Are Online vs Offline Metrics?
- offline store
- The offline store is where historical features live for training.
- Lesson 2883 — Feature Store Architecture Overview
- One-shot
- One-shot is computationally cheaper (only one retraining cycle) but often leads to larger accuracy drops, especially at high sparsity levels.
- Lesson 2669 — One-Shot vs Iterative Pruning
- One-shot prompting
- One-shot prompting is like showing someone a single map route and hoping they understand navigation principles.
- Lesson 1838 — One-Shot vs Many-Shot Trade-offs
- One-shot pruning
- One-shot pruning means you identify and remove all weights below your threshold in a single pass.
- Lesson 2669 — One-Shot vs Iterative Pruning
- one-sided label smoothing
- This is called one-sided label smoothing because you typically only smooth the "real" labels, not the fake ones.
- Lesson 1505 — Label Smoothing for GANs
- Online learning
- Online learning means your model updates incrementally with each new example (or small batch) as it arrives, adapting in real-time without needing to retrain from scratch on all historical data.
- Lesson 132 — Online Learning: Updating Models in Real-Time
- online store
- The online store is designed for real-time prediction.
- Lesson 2883 — Feature Store Architecture Overview
- OpenCLIP
- OpenCLIP is an open-source reimplementation that reproduces CLIP's results and goes further.
- Lesson 1400 — CLIP Variants and Improvements
- operators
- Airflow operators are the building blocks that define what actually gets executed in each node of your DAG.
- Lesson 2872 — Airflow Operators for ML Workflows
- Optimal strategies in some games
- Sometimes mixing actions probabilistically is actually optimal (like bluffing in poker)
- Lesson 2140 — Policies: Deterministic vs Stochastic
- optimistic initial values
- In reinforcement learning, optimistic initial values is a clever trick where you initialize all action-value estimates (Q-values) to optimistically high numbers instead of zero or realistic values.
- Lesson 2201 — Optimistic Initial Values
- Optimistic initialization
- Optimistic initialization means setting your initial Q-values deliberately higher than any realistic reward you expect to receive.
- Lesson 2193 — Optimistic Initialization
- Orchestration
- Orchestration is the "conductor" of your pipeline—it decides when and in what order pipeline steps should run, manages dependencies between steps, handles retries on failure, and coordinates the overall workflow.
- Lesson 2860 — Pipeline Orchestration vs Execution
- Ornstein-Uhlenbeck noise
- Ornstein-Uhlenbeck noise is popular because it's temporally correlated (smoother exploration than pure Gaussian).
- Lesson 2196 — Exploration in Continuous Action Spaces
- orthogonal matrix
- An orthogonal matrix is a square matrix whose columns (and rows) are orthonormal vectors—meaning each column has length 1 and is perpendicular to all other columns.
- Lesson 21 — Orthogonal Matrices and Their Properties
- Orthogonal vectors
- Orthogonal vectors are vectors that meet at right angles (90 degrees).
- Lesson 20 — Orthogonality and Orthonormal Vectors
- orthonormal basis
- An orthonormal basis is a set of orthonormal vectors that span a space—like a perfectly aligned, unit-length coordinate system.
- Lesson 20 — Orthogonality and Orthonormal Vectors
- outcome
- An outcome is a single element from the sample space—one specific result.
- Lesson 53 — Sample Spaces and Events
- Outer loop
- The real judges (outer test folds) who never saw your practice sessions
- Lesson 498 — Nested Cross-Validation for Hyperparameter Tuning
- Outer vs inner loops
- The outer loop iterates over episodes (full environment runs), while the inner loop handles individual timesteps within each episode.
- Lesson 2245 — Training Loop Structure
- outlier
- An outlier is a statistically extreme value (like a temperature reading of 500°C when typical values are 20-30°C).
- Lesson 3056 — Outlier and Anomaly Detection in Data
- over-smoothing
- This phenomenon is called over-smoothing.
- Lesson 2505 — Multiple Message Passing Layers and Depth
- overall silhouette score
- The overall silhouette score is the average across all points, ranging from -1 to +1.
- Lesson 342 — Silhouette Score
- Overlapping chunks
- Overlapping chunks means each chunk shares some tokens with its neighbors.
- Lesson 1985 — Overlapping Chunks
- Oversampling
- Oversampling means creating more copies of the minority class samples so the training set becomes more balanced.
- Lesson 539 — Resampling: Oversampling the Minority Class
P
- padding mask
- A padding mask is a binary matrix (same shape as your attention score matrix) that marks which positions are real tokens (1 or True) and which are padding (0 or False).
- Lesson 1063 — Padding Masks for Variable-Length Sequences
- Page table
- A mapping that translates logical block IDs to physical memory locations
- Lesson 2971 — Virtual Memory Concepts for LLM Serving
- Parallel processing
- Parallel processing is possible (multiple agents researching different sources)
- Lesson 2111 — Multi-Agent Systems: Motivation and Use Cases
- Parameterization
- Parameterization means externalizing these decisions into configuration files or command-line arguments.
- Lesson 2863 — Parameterization and Configuration
- parameters
- Parameters are like the final texture and taste that emerge from the baking process—they develop automatically based on what happens in the oven
- Lesson 120 — ML is Optimization, Not MagicLesson 189 — Parameters vs HyperparametersLesson 505 — What Are Hyperparameters vs Parameters
- Pareto frontier
- The Pareto frontier represents the best possible combinations—points where you can't improve fairness without losing accuracy, or vice versa.
- Lesson 3315 — Trade-offs Between Fairness and Accuracy
- Pareto optimization
- Pareto optimization is an alternative: instead of forcing a single score, identify solutions where improving one metric requires sacrificing another.
- Lesson 3101 — Multi-Task and Multi-Objective Evaluation
- Partial match
- Some systems give partial credit when boundaries overlap, even if not exact. This is less common but useful for understanding model behavior during development.
- Lesson 1294 — NER Evaluation Metrics
- Passage retrieval
- Passage retrieval is the step before span prediction.
- Lesson 1301 — Context Encoding and Passage Retrieval
- path
- A path is a sequence of edges connecting two nodes.
- Lesson 2487 — Graph Properties: Degree, Connectivity, and Paths
- Path filtering
- Path filtering is the practice of pre-screening your generated reasoning chains before applying majority voting.
- Lesson 1885 — Filtering Low-Quality Paths
- Path refinement
- Path refinement means learning from failed attempts to make smarter choices when exploring alternatives.
- Lesson 1894 — Backtracking and Path Refinement
- patience
- If performance doesn't improve for a specified number of epochs (called patience), you stop training and restore the model to its best state.
- Lesson 832 — Early Stopping Implementation
- perceptron
- A perceptron is the most basic artificial neuron, designed as a linear classifier.
- Lesson 590 — The Perceptron: A Single Artificial Neuron
- perceptron learning rule
- The perceptron learning rule is an iterative algorithm that teaches a single neuron to classify data correctly.
- Lesson 591 — Perceptron Learning Rule: Training a Single Neuron
- Performance
- The Normal Equation has time complexity O(n³) due to matrix inversion, where n is the number of features. For very large feature sets (thousands+), iterative methods like gradient descent become more efficient.
- Lesson 202 — Computing the Normal Equation in NumPy
- Performance degradation
- The final global model may perform poorly on all clients, especially underrepresented ones
- Lesson 3356 — Handling Non-IID Data
- persona
- A persona is a character or role you assign to the model that governs how it responds, not just what it responds with.
- Lesson 1855 — Defining Model Personas
- PGD
- PGD is essentially BIM with random initialization—instead of starting from the clean image, you start from a random point within the perturbation budget, then iterate.
- Lesson 3390 — Basic Iterative Method (BIM) and PGD
- pgvector
- pgvector is the most popular PostgreSQL extension for vector operations.
- Lesson 1967 — Embedding Traditional Databases: pgvector and Extensions
- phonemes
- In spoken language, phonemes are the smallest distinct units of sound that differentiate meaning.
- Lesson 2447 — Phonemes and Linguistic Units
- physical batch size
- This physical batch size is what actually fits in memory during a single forward-backward pass.
- Lesson 2783 — Effective Batch Size vs Physical Batch Size
- Physical blocks
- Actual GPU memory locations where those blocks are stored
- Lesson 2973 — Block Management and Page Tables
- Physical memory
- The actual GPU memory is divided into fixed-size pages (like apartments)
- Lesson 2971 — Virtual Memory Concepts for LLM Serving
- Physical-world adversarial examples
- Physical-world adversarial examples are designed to remain effective after undergoing transformations like printing, photography, lighting changes, viewing angles, and environmental conditions.
- Lesson 3398 — Physical-World Adversarial Examples
- Pinecone
- Pinecone is a fully managed vector database that handles infrastructure, scaling, and optimization for you.
- Lesson 1966 — Vector Database Options: Pinecone, Weaviate, Qdrant
- Pipeline changes
- Preprocessing code updates (new normalization, augmentation). The data content is identical, but the transformed features differ completely.
- Lesson 2837 — Why Data Versioning Matters in ML
- Pixel features
- Pixel features are simpler and end-to-end trainable, allowing the visual encoder to adapt to the task.
- Lesson 1385 — Region Features vs Pixel Features in VL Models
- policy
- A policy is the strategy your agent follows—it tells the agent what action to take in any given state.
- Lesson 2140 — Policies: Deterministic vs Stochastic
- Policy improvement
- Identify which actions are better than what your current policy suggests
- Lesson 2143 — Action-Value Functions: Q-Functions
- polynomial kernel
- The polynomial kernel is another way to create nonlinear decision boundaries in SVMs, but it works differently.
- Lesson 283 — Polynomial Kernel and Degree Selection
- polysemantic neurons
- Remember from lesson 3269 that neurons often represent multiple unrelated concepts simultaneously—a phenomenon called polysemantic neurons or superposition.
- Lesson 3276 — Sparse Autoencoders for Disentanglement
- Poor generalization
- The model effectively becomes smaller than intended
- Lesson 1693 — Load Balancing in MoE
- population
- Key Takeaway: A population is the complete set you want to understand; a sample is the smaller subset you actually observe.
- Lesson 75 — Population vs Sample
- Population parameters
- Population parameters are the true values (mean, variance, etc.) that describe the entire population.
- Lesson 75 — Population vs Sample
- Position becomes absolute context
- The model treats "the 10th word" differently whether it appears in a 15-word sentence or a 500-word document, even though the local context might be identical.
- Lesson 1086 — Absolute Positional Embeddings: Advantages and Limitations
- position bias
- In practice, however, position bias means the judge favors whichever output appears first (or sometimes last), regardless of actual merit.
- Lesson 3164 — Position Bias in LLM Judges
- Post-deployment validation
- Post-deployment validation is the critical monitoring period immediately after deployment where you actively watch for unexpected issues that testing missed.
- Lesson 3094 — Post-Deployment Validation
- posterior
- The posterior is the updated probability that you have the disease after seeing your symptoms.
- Lesson 329 — Bayes' Theorem and Posterior Probability
- Practical significance
- The effect matters for your business
- Lesson 3078 — Interpreting A/B Test Results
- Pre-filtering
- Apply metadata conditions first, then search vectors within that subset (more efficient, but may miss edge cases if the filtered set is small)
- Lesson 1968 — Metadata Filtering in Vector Search
- Predictable spread
- The standard deviation of sample means equals the population standard deviation divided by √n
- Lesson 81 — Central Limit Theorem
- Prediction Loss
- Prediction Loss is your usual objective (cross-entropy, MSE, etc.)
- Lesson 3311 — Regularization for Fairness
- Predictive mean
- The most likely output value, computed using the kernel's covariance between x\* and your training data
- Lesson 573 — GP Prediction: Mean and Uncertainty
- Prefix tuning
- Prefix tuning is like stationing a helpful guide on every floor who can directly assist visitors at each level.
- Lesson 1740 — Prompt Tuning vs Prefix Tuning
- Premature Conclusions
- The model reaches an answer before completing necessary reasoning steps, then backfills justification that appears complete but skips critical verification.
- Lesson 1874 — Chain-of-Thought Hallucinations and Errors
- Preserving meaning
- Preserving meaning is critical—models must avoid hallucinations or semantic drift.
- Lesson 1319 — Paraphrasing and Text Simplification
- Primacy effects
- The first experience disproportionately shapes user perception. A frustrating initial interaction might cause users to avoid the feature later, even after you've fixed the issues.
- Lesson 3081 — Long-Term Effects and Novelty Bias
- primal formulation
- The primal formulation is the original way to state the SVM problem before any mathematical transformations.
- Lesson 271 — Primal Formulation of Hard-Margin SVM
- primary metric
- Your primary metric is the single north star you're optimizing for—the metric that will determine whether variant B "wins." This should align tightly with your core business objective.
- Lesson 3073 — Choosing Evaluation Metrics for A/B Tests
- prior
- The prior is how common a disease is in the population.
- Lesson 329 — Bayes' Theorem and Posterior Probability
- Private Inference
- A model runs inside the enclave, processing user queries without the cloud provider (or even the OS) seeing the input data or outputs
- Lesson 3373 — Trusted Execution Environments
- Probabilistic outputs
- Most ML models output probabilities or confidence scores—"I'm 87% confident this is a cat"—not binary certainties.
- Lesson 122 — ML Models as Approximations
- Probability Flow ODE
- The Probability Flow ODE is a remarkable discovery: there exists a deterministic ordinary differential equation that produces exactly the same marginal distributions as the stochastic SDE, but without any randomness.
- Lesson 1561 — Probability Flow ODE
- process group
- The rank is a unique ID for each process (0, 1, 2, ...), the world size is the total number of processes, and a process group is the communication channel connecting them all.
- Lesson 2794 — Distributed Process Groups and Ranks
- Profiling
- Profiling means examining what autograd is actually tracking—checking if gradients exist where expected and understanding why they might be missing.
- Lesson 800 — Autograd Profiling and Common Pitfalls
- projection head
- Instead, we add a small MLP—called a projection head—that transforms h into z, and compute the InfoNCE loss on z.
- Lesson 2551 — Projection Head Design and Representation Quality
- projects
- Think of W&B's organizational system like a filing cabinet: projects are drawers, groups are folders within drawers, tags are colored sticky notes, and notes are detailed labels.
- Lesson 2816 — W&B Run Management and Organization
- Prompt engineering as defense
- Prompt engineering as defense means architecting your system prompt with structural boundaries that make it harder for user input to masquerade as system instructions.
- Lesson 3423 — Defense: Prompt Engineering Against Injection
- prompt template
- A prompt template is a reusable prompt structure with placeholders—special markers that indicate where dynamic content should be inserted.
- Lesson 1847 — Prompt Templates and Placeholders
- Prompt tuning
- Prompt tuning is like giving special instructions to visitors at the lobby entrance.
- Lesson 1740 — Prompt Tuning vs Prefix Tuning
- Provenance tracking
- Provenance tracking means recording the complete lineage of your data:
- Lesson 1642 — Documenting and Reproducing Data Pipelines
- Proxy metrics
- Proxy metrics are measurable signals available immediately that correlate with model quality.
- Lesson 3018 — Proxy Metrics for Real-Time Monitoring
- pseudo-residuals
- Fit a tree to residuals: Train a new tree to predict these errors (these are called pseudo-residuals)
- Lesson 312 — Gradient Boosting for Regression
- Public disclosure
- Public disclosure means revealing a vulnerability openly—through blog posts, conference talks, social media, or research papers—before it's fully patched or acknowledged.
- Lesson 3526 — Public Disclosure Decisions
- Public trust
- Public trust is critical to your product's success
- Lesson 3325 — External and Third-Party Audits
Q
- Q-functions
- That's where action-value functions, commonly called Q-functions, come in.
- Lesson 2143 — Action-Value Functions: Q-FunctionsLesson 2148 — Action-Value Functions (Q-Functions)
- Q-learning
- Q-learning is like studying the optimal racing line in theory, even while you drive conservatively.
- Lesson 2178 — Q-Learning vs SARSA: Key Differences
- Qdrant
- Qdrant is a lightweight, Rust-based vector database emphasizing performance and simplicity.
- Lesson 1966 — Vector Database Options: Pinecone, Weaviate, Qdrant
- quadratic form
- A quadratic form is an expression that looks like x^T A x, where:
- Lesson 26 — Quadratic Forms
- Quality preservation
- A well-trained encoder (from VAE training) captures the semantically important features while discarding imperceptible details. The diffusion model learns to generate in this meaningful, compressed space where the structure is already more abstract and easier to model.
- Lesson 1565 — From Pixel Space to Latent Space Diffusion
- Quantiles
- Quantiles are the general term for these division points.
- Lesson 78 — Percentiles and Quantiles
- quantization error
- This loss is called quantization error, and it's determined by how you handle numbers that fall between integer steps.
- Lesson 2627 — Quantization Error and Rounding
- query
- Think of it like searching a library: your query is "books about transformers," each book's key is its title and topic tags, and the value is the book's actual content.
- Lesson 1051 — Query, Key, Value: The Three Vectors
- Query Set
- Unlabeled examples from the same classes that the model must classify after "seeing" the support set. This is your test set for evaluating how well the model generalized from those few support examples.
- Lesson 2585 — Support Set vs Query Set
- Query transformation
- Query transformation means converting a user's natural language question into a machine-executable query format.
- Lesson 2021 — Query Transformation for Structured Data
- Question Answering
- The context/passage is encoded separately from generating the answer
- Lesson 1102 — Encoder-Decoder vs Decoder-Only Trade-offs
- Queue depth limits
- Reject new requests when the queue is full (fail fast)
- Lesson 3007 — Request Queuing and Priority Management
R
- RAG
- RAG is like giving someone a research library and teaching them to look things up on demand
- Lesson 1953 — RAG vs Fine-Tuning: When to Use Each
- Random undersampling
- Random undersampling is fastest but risks losing informative samples.
- Lesson 542 — Resampling: Undersampling the Majority Class
- random variable
- A random variable is simply a numerical outcome of a random process.
- Lesson 58 — Random Variables: Discrete and Continuous
- rank
- The rank is a unique ID for each process (0, 1, 2, ...), the world size is the total number of processes, and a process group is the communication channel connecting them all.
- Lesson 2717 — Process Groups and InitializationLesson 2794 — Distributed Process Groups and Ranks
- receptive field
- A receptive field is the region of the input image that influences the activation of a single neuron in a particular CNN layer.
- Lesson 879 — What is a Receptive Field?
- reconstruction loss
- The reconstruction loss is your primary training signal.
- Lesson 1435 — Training Dynamics and Convergence
- Recurrent connections
- Standard dropout can disrupt temporal dependencies in RNNs. Specialized variants like recurrent dropout are needed instead.
- Lesson 750 — When Dropout Helps and When It Doesn't
- Red team testing
- Red team testing is the practice of deliberately trying to break your model's alignment—finding prompts that cause harmful outputs despite your constitutional principles.
- Lesson 1826 — Iterative Refinement and Red Team Testing
- reduce
- Reduce is less common alone because in data parallelism you typically need the combined result everywhere, not just on one worker.
- Lesson 2721 — Broadcast and Reduce Operations
- Reduce bias
- Judges reasoning about one clear criterion are less likely to conflate issues
- Lesson 3167 — Multi-Aspect Evaluation with LLM Judges
- Reduced confusion
- The model knows exactly what information to use and what operation to perform
- Lesson 1843 — Context vs. Task Separation
- Reduces co-adaptation
- The network can't rely on any single layer always being present
- Lesson 748 — Stochastic Depth
- Reduces noise
- Small fluctuations within a bin are ignored
- Lesson 441 — Binning and Discretization Techniques
- Redundancy reduction term
- Pushes off-diagonal elements toward 0 (dimensions are decorrelated)
- Lesson 2565 — Barlow Twins: Redundancy Reduction
- reference model
- In Direct Preference Optimization (DPO), the reference model is a frozen copy of your starting policy (typically the SFT model).
- Lesson 1808 — The Reference Model in DPO
- Regularization
- Regularization is the practice of adding a penalty for model complexity directly into your loss function.
- Lesson 223 — Introduction to Regularization
- Regularization effect
- The noise from batch statistics acts like a mild regularizer
- Lesson 873 — Batch Normalization in CNNs
- regularization path
- A regularization path is a visualization that shows how each feature's coefficient changes as you sweep through different regularization strengths.
- Lesson 232 — Regularization Paths
- Regularization strength
- Generally provides finer-grained regularization since individual weights are masked
- Lesson 747 — DropConnect and Weight Dropping
- Relative time distances
- The gap between observations matters (1 minute vs 1 week)
- Lesson 2417 — Transformers for Time Series Forecasting
- reliability diagram
- A calibration plot (also called a reliability diagram) does exactly this check for your ML model's probability predictions.
- Lesson 489 — Calibration Plots and Reliability Diagrams
- Replay Buffer Size
- Think of this as your agent's memory capacity. A larger buffer (1M transitions) provides more diverse experiences but uses more RAM and may include outdated policies. Smaller buffers (100K) update faster but risk overfitting to recent experiences. Prioritized replay amplifies this sensitivity—if your buffer is too small, you'll replay the same "important" transitions obsessively.
- Lesson 2235 — Hyperparameter Sensitivity in DQN Variants
- Reproducibility
- The hash uniquely identifies the exact dataset version used
- Lesson 2839 — Content-Addressable Storage for Data
- Resampling
- Resampling is the process of converting data from one temporal resolution to another—like converting hourly temperature readings into daily averages, or filling in monthly sales data to get weekly estimates.
- Lesson 2394 — Resampling and Frequency Conversion
- Reservation
- These tokens are added to the vocabulary explicitly and assigned fixed IDs, often at the beginning or end of the vocabulary range.
- Lesson 1648 — Handling Special Tokens
- residual
- A residual is simply the error your model makes on each data point: the actual value minus the predicted value.
- Lesson 190 — Residuals and Prediction ErrorsLesson 527 — Residual Analysis for Regression
- residual block
- A residual block is the concrete implementation of skip connections in ResNet.
- Lesson 904 — The Residual Block Architecture
- ResNet-50
- A residual block might have [64, 64, 256] filters across its convolutions
- Lesson 910 — ResNet Family: 18, 34, 50, 101, 152Lesson 911 — Wide Residual Networks (WRN)
- Resource-constrained planning
- Resource-constrained planning means designing agent behavior that achieves goals while staying within hard limits on:
- Lesson 2093 — Resource-Constrained Planning
- Result Storage and Display
- Computed metrics are stored with metadata (timestamp, model description, hyperparameters) and displayed on a public leaderboard, often with filtering, sorting, and historical tracking capabilities.
- Lesson 3125 — Leaderboards and Evaluation Infrastructure
- Retrieval decision making
- Retrieval decision making means using the LLM itself to classify whether a query requires external context or can be answered directly from its parametric knowledge.
- Lesson 2046 — Retrieval Decision Making
- Reveal patterns
- Systematic residuals indicate your model is missing something important
- Lesson 190 — Residuals and Prediction Errors
- Reversibility
- Reversibility means your tokenization process preserves enough information to convert tokens back to text exactly as it was.
- Lesson 1247 — Reversibility and Detokenization
- Reward hacking
- Specification gaming (also called reward hacking) occurs when a model discovers and exploits these loopholes, achieving high measured performance while failing at the true underlying goal.
- Lesson 1791 — The Trust Region ConstraintLesson 3426 — Specification Gaming and Reward HackingLesson 3431 — The Scalable Oversight Problem
- Richer generation context
- The LLM sees the full picture, not isolated fragments
- Lesson 1994 — Parent-Child Chunking
- Right to know
- Individuals must be informed when significant decisions are automated
- Lesson 3505 — Algorithmic Transparency and Explainability Requirements
- risk taxonomy
- A risk taxonomy is a structured classification system—think of it as organizing a messy toolbox into labeled drawers.
- Lesson 3531 — Risk Identification and Taxonomy
- Robustness
- The model doesn't overfit to one specific tokenization pattern
- Lesson 1263 — Subword Regularization
- ROC Curve
- The ROC Curve is a graph that shows how well your classifier performs at every possible threshold setting.
- Lesson 480 — Receiver Operating Characteristic (ROC) Curve
- Role and persona assignment
- Role and persona assignment means telling the model who it should act as when generating a response.
- Lesson 1848 — Role and Persona Assignment
- Role-based agent specialization
- Role-based agent specialization means deliberately designing agents with focused capabilities, knowledge, and responsibilities.
- Lesson 2114 — Role-Based Agent Specialization
- Roles
- A "researcher" agent retrieves information while a "writer" agent drafts responses
- Lesson 2111 — Multi-Agent Systems: Motivation and Use Cases
- rollback mechanism
- A rollback mechanism is your emergency exit: a predefined procedure to instantly restore the last known good model version before issues impact more users.
- Lesson 3090 — Rollback Mechanisms
- Rollout generation
- Rollout generation means sampling complete response sequences from your current language model (the policy) given various prompts, then collecting the rewards for each of those generations.
- Lesson 1796 — Rollout Generation and Experience Collection
- Routing
- Routing means using the question itself to decide which source(s) to query.
- Lesson 2051 — Routing to Multiple Knowledge Sources
- row
- Each row represents one query position (what's currently "asking")
- Lesson 1053 — The Attention Score Matrix
- Rows
- Query positions (what the model is currently processing)
- Lesson 1059 — Understanding Attention Weight Visualization
S
- SAC
- SAC is also stable but requires careful tuning of the temperature parameter (α), which controls the exploration-exploitation trade-off.
- Lesson 2324 — SAC vs TD3: When to Use Which
- Saddle Point
- A point that's a minimum in some directions but a maximum in others—like a mountain pass or a horse saddle.
- Lesson 95 — Local vs Global Optima
- saliency map
- A saliency map is a pixel-level visualization that highlights the spatial regions of an input image that had the greatest impact on a neural network's output.
- Lesson 3231 — What Are Saliency Maps?
- sample
- Key Takeaway: A population is the complete set you want to understand; a sample is the smaller subset you actually observe.
- Lesson 75 — Population vs Sample
- Sample Efficiency
- PPO is on-policy, requiring fresh experience for each update. In environments where data collection is expensive (real robots, expensive simulations), off-policy methods like SAC may be more practical.
- Lesson 2314 — PPO in Practice: Success Stories and Limitations
- Sample quality
- A good schedule preserves image structure in early steps
- Lesson 1526 — Variance Schedule: Controlling Noise Addition
- Sample statistics
- Sample statistics are the values we calculate from our sample data.
- Lesson 75 — Population vs Sample
- Sampling
- You can generate new data by simply sampling z ~ N(0, I) and passing it through the decoder. Because the latent space is trained to match this distribution, random samples will decode into plausible outputs.
- Lesson 1447 — Why the Prior Matters
- sampling distribution
- A sampling distribution is the probability distribution of these sample statistics (like the mean, variance, or standard deviation) across many possible samples.
- Lesson 82 — Sampling Distributions
- SARSA
- SARSA is like learning from your actual driving experience, including all your cautious decisions and mistakes.
- Lesson 2178 — Q-Learning vs SARSA: Key Differences
- Saturation effects
- Neural networks use activation functions like ReLU or sigmoid. When neurons saturate (flatten out), their gradients become zero or near-zero, even if those features are genuinely important. The gradient "dies" not because the feature doesn't matter, but because the neuron has hit its flat region.
- Lesson 3234 — Why Raw Gradients Are Noisy
- Scalability
- The single-stack design scales efficiently as you add parameters
- Lesson 1200 — Decoder-Only Design: Why GPT Diverged from BERT
- Scale and coverage
- A single research team can't test every edge case. Crowdsourcing naturally explores long-tail queries and niche domains.
- Lesson 3177 — Chatbot Arena and Community Evaluation
- Scale-independent evaluation
- Scale-independent evaluation means you can compare models across different datasets or target ranges.
- Lesson 473 — Mean Absolute Percentage Error (MAPE)
- Scattered attention
- Either the model is confused, or the task genuinely requires broad context integration.
- Lesson 1059 — Understanding Attention Weight Visualization
- Scheduling and triggers
- Scheduling and triggers are the mechanisms that determine when your DAG executes.
- Lesson 2864 — Scheduling and Triggers
- Schema compliance
- The JSON may be valid but not match your desired structure
- Lesson 1913 — Native JSON Mode in Modern LLMs
- score function
- A score function is simply the gradient of the log-probability density with respect to the input data.
- Lesson 1553 — Score Functions and the Score Matching Objective
- score matching
- At its core, score matching is about learning the score function—the gradient of the log probability of your data distribution.
- Lesson 1535 — Connection to Score Matching
- Score near +1
- Point is well-matched to its cluster and far from others (great!)
- Lesson 342 — Silhouette Score
- search space
- The search space is the complete set of possible values you allow each hyperparameter to take when tuning your model.
- Lesson 506 — The Hyperparameter Search Space
- Seasonal decomposition
- Seasonal decomposition is the process of separating that chord back into its individual notes: the long-term trend (where things are heading overall), the repeating seasonal pattern (predictable cycles like weekly or yearly fluctuations), and the residual or noise (random, unexplained variation).
- Lesson 2403 — Seasonal Decomposition
- Secondary models
- A specialized model scores factual accuracy or safety
- Lesson 1943 — External Validators in Refinement Loops
- Secure aggregation
- Secure aggregation is a cryptographic protocol that allows a central server to compute the sum of inputs from many users while mathematically guaranteeing that no individual input is revealed—even to the server itself.
- Lesson 3368 — Secure Aggregation Protocol
- secure enclave
- TEEs provide an isolated execution space called a secure enclave where sensitive computations can run.
- Lesson 3373 — Trusted Execution Environments
- Security severity
- Targeted attacks are often more dangerous. Making a spam filter classify one message as "not spam" is targeted; making it misclassify randomly is untargeted.
- Lesson 3379 — Targeted vs Untargeted Attacks
- Self-Adversarial Training
- The network slightly modifies images to fool itself, then learns from those "attacks"
- Lesson 965 — YOLOv4 and YOLOv5: Speed and Accuracy Advances
- Self-attention
- Self-attention is like understanding grammar and word relationships within the English text itself
- Lesson 1078 — Cross-Attention vs. Self-Attention Heads
- Semantic segmentation
- Labels every pixel with its class ("this pixel is sky, that pixel is tree, this pixel is road")
- Lesson 975 — What Is Semantic Segmentation
- Semi-linear structure
- The diffusion ODE has a particular mathematical form that allows efficient high-order approximations
- Lesson 1602 — DPM-Solver and ODE Solvers
- sensitive features
- Protected attributes (also called sensitive features) are characteristics of individuals that should not—legally or ethically—influence certain decisions.
- Lesson 3294 — Protected Attributes and Sensitive Features
- Sentiment analysis
- The end of a sentence often flips meaning ("I thought it was good... NOT!")
- Lesson 1024 — Bidirectional LSTMs and GRUs
- Separation by masking
- The network learns to predict a multiplicative mask for each source. When you multiply the mixture spectrogram by the vocal mask, you get (mostly) just the vocals.
- Lesson 2481 — Audio Source Separation
- shape
- The shape is a tuple showing the length along each dimension.
- Lesson 151 — Array Shapes and Dimensions in ML
- Shared Context
- Shared Context refers to common knowledge all agents can access: the current task state, goals, constraints, and environmental observations.
- Lesson 2120 — Shared Context and Memory in Multi-Agent Systems
- Shared Memory
- A small (48-164KB per streaming multiprocessor), ultra-fast scratchpad explicitly controlled by kernels. Custom CUDA kernels use shared memory to stage frequently-accessed data, reducing VRAM trips. This is critical for optimized operations like flash attention.
- Lesson 2120 — Shared Context and Memory in Multi-Agent SystemsLesson 2935 — Understanding GPU Memory Hierarchy for Inference
- Sharpening
- A low temperature is applied to the teacher's softmax outputs (like we saw in contrastive learning), making the predictions more confident and peaked. The student, however, uses a higher temperature, creating softer targets.
- Lesson 2567 — DINO: Self-Distillation with No Labels
- Short-term memory
- Short-term memory (working memory) is the agent's current context—the immediate conversation, the task at hand, and recent observations from the environment.
- Lesson 2097 — Short-Term vs Long-Term Memory in Agents
- Short-term optimization
- Short-term optimization means telling clients their form is perfect and they can skip hard exercises—instant satisfaction, five-star ratings.
- Lesson 3445 — Short-Term vs Long-Term Alignment
- SiLU
- Unlike ReLU which abruptly cuts off negative values, Swish (also called SiLU - Sigmoid Linear Unit) creates a smooth, self-gated activation by multiplying the input by its own sigmoid.
- Lesson 660 — Swish and SiLU: Self-Gated Activations
- Simplicity
- Absolute encodings are conceptually simple—each position has a fixed code. This makes implementation and debugging easier.
- Lesson 1086 — Absolute Positional Embeddings: Advantages and Limitations
- SimSiam
- SimSiam is the most memory-efficient: no momentum encoder, no extra memory banks—just stop-gradient.
- Lesson 2570 — Comparing Non-Contrastive Approaches
- skip-gram architecture
- While you've learned that the distributional hypothesis tells us "words that appear in similar contexts have similar meanings," the skip-gram architecture is one specific way to learn those dense word embeddings.
- Lesson 1119 — Word2Vec: Skip-gram Architecture
- Slice-based evaluation
- Slice-based evaluation means systematically measuring model performance on meaningful subsets (slices) of your data—defined by features, combinations of features, or other criteria—to uncover hidden disparities.
- Lesson 3127 — What is Slice-Based Evaluation?
- Slower convergence
- The algorithm takes many more communication rounds to reach acceptable performance
- Lesson 3356 — Handling Non-IID Data
- small margin
- A small margin means your decision boundary is close to some training points—it's confident but fragile
- Lesson 268 — The Concept of Margin
- Small negative values
- Small negative values (close to zero) are usually statistical noise—treat them as unimportant features.
- Lesson 3201 — Interpreting Negative Importance Values
- Smoothness
- Unlike ReLU's sharp corner at zero, GELU is differentiable everywhere, which can improve gradient flow
- Lesson 659 — GELU: Gaussian Error Linear Units
- Soft targets
- Soft targets are the full probability distribution output by the teacher model—capturing not just what the teacher predicts, but how confident it is and which alternative classes seemed plausible.
- Lesson 2680 — Soft Targets and Temperature Scaling
- Softmax loss on pairs
- Classify whether sentence pairs are similar
- Lesson 1972 — Sentence Transformers Architecture
- Span
- Span is the collection of all possible destinations you can reach using linear combinations (addition and scalar multiplication) of your vectors.
- Lesson 10 — Linear Independence and Span
- span prediction
- This is called span prediction, and it works by predicting two things: where the answer starts and where it ends.
- Lesson 1300 — Span Prediction with BERT
- Span-based
- Answers are always continuous sequences from the context
- Lesson 1298 — Extractive QA Fundamentals
- Sparse problems
- Many machine learning problems have sparse solutions (most coefficients are zero), and coordinate descent can efficiently identify and update only the relevant variables
- Lesson 109 — Coordinate Descent
- Sparse Rewards
- PPO struggles when feedback is rare and delayed. Without frequent reward signals, advantage estimates become unreliable, leading to slow or failed learning.
- Lesson 2314 — PPO in Practice: Success Stories and Limitations
- Sparsity
- Weighted adjacency matrices remain sparse; computational efficiency is preserved
- Lesson 2507 — Handling Directed and Weighted Graphs
- sparsity constraint
- The sparsity constraint is crucial—it forces the model to use only a few features at a time, preventing it from learning "cheating" distributed representations.
- Lesson 3276 — Sparse Autoencoders for Disentanglement
- sparsity target
- Your sparsity target is simply the final percentage of weights you want to prune.
- Lesson 2670 — Pruning Schedules and Sparsity Targets
- Spectral envelope
- The overall frequency distribution that identifies vowels and consonants
- Lesson 2446 — Speech Signal Fundamentals
- Spectral normalization
- Spectral normalization is a technique that normalizes each weight matrix in your discriminator by dividing it by its spectral norm—the largest singular value of that matrix.
- Lesson 1508 — Spectral Normalization
- Speed
- Greedy decoding is extremely fast—just one forward pass per token, no search overhead.
- Lesson 336 — Naive Bayes Advantages and LimitationsLesson 1191 — Greedy Decoding
- Speed improvements
- The denoising network (U-Net) processes smaller tensors, meaning:
- Lesson 1575 — Computational Benefits of Latent Diffusion
- spherical
- Think of it like buying furniture: full covariance is custom-made (expensive, perfect fit), diagonal is adjustable (width/height independent), spherical is one-size (cheap, fast), and tied means all rooms get identical furniture (consistent but less flexible).
- Lesson 371 — Covariance Structure Constraints
- Stability
- The model won't suddenly start generating nonsense
- Lesson 1666 — Training Strategies for Long ContextLesson 1789 — PPO Overview: Policy Optimization for LLMs
- Stable Learning
- Low-resolution patterns are easier to learn first
- Lesson 1485 — Progressive Growing of GANs (ProGAN)
- Stage 2
- Gradients + optimizer states are your bottleneck; balanced approach
- Lesson 2748 — Memory vs Communication Tradeoffs
- Staging
- Model passed initial tests and is ready for validation in a production-like environment
- Lesson 2832 — Model Staging and Promotion
- state
- A state is a snapshot of the environment at a specific moment.
- Lesson 2083 — Planning in AI Agents: Problem FormulationLesson 2134 — States, Actions, and State Spaces
- state space
- The state space is the set of all possible states the environment can be in.
- Lesson 2134 — States, Actions, and State Spaces
- States
- Each cell in the grid is a state (e.g., position (2,3))
- Lesson 2145 — Gridworld: A Classic MDP Example
- Static quantization
- Static quantization is like a standardized recipe where every measurement is predetermined, including seasonings—faster execution but requires testing batches first to get it right.
- Lesson 2632 — Dynamic vs Static Quantization
- Static shape handling
- Static shape handling means your model is compiled and optimized assuming inputs always have the same dimensions—for example, images always 224×224 or sequences always length 512.
- Lesson 2952 — Static vs Dynamic Shape Handling
- Static thresholds
- Static thresholds are simple but brittle: "Alert if error rate > 5%." They break when normal behavior changes (weekends vs weekdays, seasonal patterns).
- Lesson 3023 — Alerting Strategies and Thresholds
- Stationarity
- Stationarity means your time series has constant statistical properties over time: specifically, constant mean, variance, and autocorrelation (how values relate to their own past).
- Lesson 2386 — Stationarity and Why It Matters
- Statistical power
- Statistical power is critical (detecting small performance differences)
- Lesson 3119 — Size vs Quality Tradeoffs
- Statistical significance
- The effect is unlikely due to chance (p-value)
- Lesson 3078 — Interpreting A/B Test Results
- Steps
- Steps are individual optimizer updates (batches processed).
- Lesson 1708 — Training Duration and Convergence
- stochastic noise
- This stochastic noise is similar to dropout—it prevents the network from relying too heavily on exact activation values and forces it to learn more robust features.
- Lesson 764 — Normalization as Regularization
- Stochastic regularization
- The probabilistic weighting acts as implicit regularization
- Lesson 659 — GELU: Gaussian Error Linear Units
- Straightforward training
- The model simply learns to predict the next token given all previous tokens
- Lesson 1186 — Left-to-Right vs Bidirectional Context
- Stratified K-Fold
- Stratified K-Fold is a smarter version of K-Fold that preserves the class distribution in every fold.
- Lesson 494 — Stratified K-Fold for Classification
- Streaming support
- Streaming support is gRPC's superpower: you can stream inputs for online learning scenarios, stream outputs for generated text/images, or both simultaneously — impossible with basic REST.
- Lesson 2905 — gRPC for High-Performance Serving
- Stride
- Stride is the parameter that controls how far the filter shifts with each step—both horizontally and vertically.
- Lesson 855 — Stride: Controlling Step Size
- stride of 1
- A stride of 1 means the filter moves one pixel at a time, computing an output value at nearly every possible position.
- Lesson 855 — Stride: Controlling Step Size
- stride of 2
- A stride of 2 means the filter jumps two pixels with each step, effectively skipping positions.
- Lesson 855 — Stride: Controlling Step Size
- Structured vs Unstructured
- Unstructured pruning (removing individual weights) offers flexibility but requires specialized hardware to achieve speedups. Structured pruning (removing entire channels or layers) is hardware-friendly but may harm accuracy more.
- Lesson 2666 — Why Prune: Benefits and Trade-offs
- Subjectivity
- Preferences often depend on subjective cultural context, personal values, or expertise. What one annotator considers "helpful" another might find "verbose." Aggregating these views is challenging.
- Lesson 1817 — Limitations of Human Feedback and Motivation for RLAIF
- Subpopulation disparities
- A fraud detector might excel on common transaction types but fail on rare, high-value cases
- Lesson 3128 — Why Aggregate Metrics Hide Problems
- Success rate
- Success rate is the primary metric: the percentage of inputs where the adversarial perturbation caused misclassification.
- Lesson 3400 — Evaluating Attack Success and Perturbation Budgets
- Successive Halving
- Successive Halving is a smarter approach: start by training many configurations with a small budget (few iterations, small data subset), then progressively eliminate the worst performers and give more resources only to the promising ones.
- Lesson 513 — Successive Halving and Early Stopping
- Sufficient for many tasks
- For most language and vision tasks, knowing "this is the 5th token" provides enough positional information for the model to learn meaningful patterns.
- Lesson 1086 — Absolute Positional Embeddings: Advantages and Limitations
- Sum
- Sum is theoretically most powerful because it preserves neighbor count information.
- Lesson 2503 — Aggregation Functions: Mean, Max, Sum
- Summarization
- The full document must be encoded before creating a condensed version
- Lesson 1102 — Encoder-Decoder vs Decoder-Only Trade-offs
- Superior accuracy
- The model attends across both inputs, capturing nuanced relevance signals
- Lesson 2006 — Bi-Encoder vs Cross-Encoder Trade-offs
- Support Set
- The few labeled examples (e.g., 5 images per class) used to adapt or condition your model for this specific task
- Lesson 2585 — Support Set vs Query SetLesson 2606 — The Meta-Learning Problem Formulation
- Swapping
- Swapping is the gold standard: always evaluate each pair twice with reversed order, then aggregate results (e.g., count a win only if the model wins in both positions, or average the scores).
- Lesson 3164 — Position Bias in LLM Judges
- Synapses
- Synapses are the connection points where signals pass between neurons.
- Lesson 589 — The Biological Neuron: Inspiration for Artificial Networks
- Synthetic request injection
- Synthetic request injection is the core technique: before marking an instance "ready," send dummy inference requests through the pipeline.
- Lesson 3009 — Model Warmup and Cold Start Optimization
- System
- Think of it like a play: the system is the director's notes (invisible to the audience but guiding everything), the user is the audience asking questions, and the assistant is the actor performing on stage.
- Lesson 1232 — Instruction Format and Template DesignLesson 1752 — Instruction Format and TemplatesLesson 1854 — System vs User vs Assistant Messages
- system prompt
- A system prompt is a special instruction block sent to an LLM that sets persistent context, defines the model's persona, and establishes behavioral constraints—all before the user's actual question or task arrives.
- Lesson 1853 — What Are System Prompts?
T
- t-SNE
- t-SNE is stochastic—run it twice, get different results (though patterns remain similar).
- Lesson 403 — UMAP vs t-SNE: Comparative Analysis
- Tabular data
- Often the features are already interpretable (age, income), but may need discretization (converting continuous values into bins like "high income" vs "low income")
- Lesson 3223 — Interpretable Representations
- tags
- Think of W&B's organizational system like a filing cabinet: projects are drawers, groups are folders within drawers, tags are colored sticky notes, and notes are detailed labels.
- Lesson 2816 — W&B Run Management and Organization
- target network
- A target network is a separate copy of your Q-network that generates the target values in your loss function.
- Lesson 2211 — Target Networks for StabilityLesson 2223 — Target Network: Stabilizing Q-Learning
- target statistics
- CatBoost uses a clever technique called target statistics (also known as greedy target-based statistics).
- Lesson 318 — CatBoost: Handling Categorical Features Natively
- Task pattern
- The transformation rule you want applied
- Lesson 1832 — Introduction to Few-Shot Prompting
- task-specific head
- A task-specific head is just a small neural network (often a single linear layer) that you attach on top of BERT to map this [CLS] representation to your specific classification problem.
- Lesson 1174 — Task-Specific Heads for Classification
- task-specific system prompts
- While generic system prompts define general behavior, task-specific system prompts are precision instruments tailored to excel at particular activities.
- Lesson 1859 — Task-Specific System Prompts
- Tasks
- Tasks are the individual units of work, like loading data or training a model.
- Lesson 2875 — Prefect Architecture and Task API
- TD approach
- After driving one block, estimate remaining time based on your current belief. Less noisy (one block is more predictable), but initially wrong if your beliefs are off.
- Lesson 2173 — TD vs Monte Carlo: Bias-Variance Tradeoff
- TD3
- TD3 is also sample-efficient but may require more samples in sparse-reward environments where exploration is critical.
- Lesson 2324 — SAC vs TD3: When to Use Which
- Teacher forcing
- Teacher forcing means using the ground truth token from the training data as the decoder's input at each time step, instead of the decoder's own prediction.
- Lesson 1029 — Teacher Forcing in Training
- Tecton
- Tecton is an enterprise-grade, managed platform built by Feast's original creators.
- Lesson 2890 — Feature Store Tools: Feast, Tecton, and Alternatives
- temperature
- In the InfoNCE loss you just learned, there's a small but mighty parameter called temperature (τ, tau) that sits in the denominator of the softmax.
- Lesson 2538 — Temperature in Contrastive Loss
- Temporal and geographic slicing
- Temporal and geographic slicing means deliberately splitting your evaluation data by time windows and location attributes to expose these hidden weaknesses.
- Lesson 3133 — Temporal and Geographic Slices
- Temporal Modeling
- Temporal Modeling is the heart of video understanding—learning which frames matter and how they relate sequentially.
- Lesson 995 — Video Understanding Tasks
- Temporal patterns
- The rhythm and duration of sounds that distinguish phonemes (basic speech units like "p" vs "b")
- Lesson 2446 — Speech Signal Fundamentals
- tensor
- A tensor is simply a multidimensional array—a container for numbers arranged in a grid with one or more dimensions.
- Lesson 775 — What is a Tensor?
- Tensor core usage
- Specialized hardware for matrix operations is more energy-efficient per operation than standard CUDA cores
- Lesson 3469 — GPU Power Consumption and Efficiency
- TensorFlow Model Analysis
- TensorFlow Model Analysis is the industry-standard library for slice-based evaluation.
- Lesson 3136 — Tools and Workflows for Slice-Based Analysis
- terminal state
- In reinforcement learning, a terminal state is where an episode ends—the game is over, the agent reached the goal, or failed catastrophically.
- Lesson 2217 — Handling Terminal States
- Testable
- You should be able to apply the principle to any model output and get a clear yes/no answer. "Choose the response that is more helpful" is testable; "be better" is not.
- Lesson 1823 — Writing and Selecting Constitutional Principles
- text degeneration
- This is called text degeneration or repetition.
- Lesson 1323 — Repetition and Degeneration Problems
- text encoder
- A text encoder is a pretrained language model (like CLIP's text encoder or T5) that transforms your text prompt into a rich numerical embedding—a vector that captures the semantic meaning of your words.
- Lesson 1590 — Text Encoder Integration
- Think of it this way
- Just as VAEs use ELBO to bridge exact likelihood (intractable) with a trainable bound, DDPMs use ELBO to connect the ideal reverse diffusion process (unknown) with your learned denoising network. The simplified loss is just the most practical form of this bound for training.
- Lesson 305 — Bagging for Other Base LearnersLesson 1547 — Connection to Variational Lower Bound
- Threat modeling
- Threat modeling is the structured process of anticipating how your language model could be attacked, misused, or fail—before those problems emerge in production.
- Lesson 3448 — Threat Modeling for Language Models
- Threshold adjustment
- Threshold adjustment means changing that cutoff point.
- Lesson 545 — Threshold Adjustment for Imbalanced Data
- Threshold optimization
- Threshold optimization means setting different thresholds for different protected groups to satisfy fairness criteria.
- Lesson 3312 — Threshold Optimization
- Threshold-based secret sharing
- Threshold-based secret sharing is the key.
- Lesson 3371 — Dropout Resilience in Secure Aggregation
- tied
- Think of it like buying furniture: full covariance is custom-made (expensive, perfect fit), diagonal is adjustable (width/height independent), spherical is one-size (cheap, fast), and tied means all rooms get identical furniture (consistent but less flexible).
- Lesson 371 — Covariance Structure Constraints
- Tiered Decision Systems
- Routine, low-risk cases are automated; medium-risk cases get human review; high-risk cases require multi-person approval.
- Lesson 3491 — Human-in-the-Loop Design Patterns
- Time
- Training passes through all layers are slow, especially for models with billions of parameters
- Lesson 1701 — What Full Fine-Tuning Means for LLMs
- Time-varying covariates
- Time-varying covariates are processed alongside the target sequence, often through separate pathways that merge with temporal representations
- Lesson 2421 — Handling Covariates and External Features
- Timeout or resource exhaustion
- An action takes too long or hits limits
- Lesson 2090 — Dynamic Replanning and Error Recovery
- Tokenization
- Tokenization is the process of breaking down raw text into smaller units called tokens—which could be words, subwords, or even individual characters—and mapping each token to a unique numerical identifier.
- Lesson 1237 — What Is Tokenization and Why It Matters
- Tool execution errors
- A function returns an error code or exception
- Lesson 2090 — Dynamic Replanning and Error Recovery
- tool registry
- An action space is the complete set of operations an agent can perform—its "toolbox." A tool registry is the system that catalogs these tools with structured descriptions so the agent knows what's available and how to use each one.
- Lesson 2062 — Action Space and Tool Registry
- tool schema
- A tool schema is a structured specification that describes:
- Lesson 2072 — Tool Schema Definition
- top-k filtering
- This is called top-k filtering or k-nearest neighbors (k-NN) selection.
- Lesson 2361 — Neighborhood Selection and Top-K Filtering
- Trade-off
- Training hundreds of architectures across many generations is computationally expensive, though techniques like weight inheritance and early stopping help.
- Lesson 2697 — Evolutionary Algorithms for NAS
- Traditional security vulnerabilities
- Traditional security vulnerabilities are the familiar weaknesses in software: SQL injection, buffer overflows, authentication bypass, insecure APIs, or exposed credentials.
- Lesson 3522 — Security Vulnerabilities vs. AI-Specific Risks
- Trained on controlled tasks
- Synthetic data where ground truth is known (e.g., modular arithmetic, simple classification)
- Lesson 3267 — Toy Models for Mechanistic Analysis
- Training
- Training is like building a factory: it's expensive, energy-intensive, and happens once (or infrequently when you retrain).
- Lesson 947 — Intersection over Union (IoU)Lesson 3468 — Measuring ML Energy ConsumptionLesson 3471 — Training vs Inference Environmental Costs
- Training instability
- The agent might unlearn good behaviors due to noisy updates
- Lesson 2255 — Variance in Policy Gradients
- training-serving skew
- This training-serving skew is one of the most insidious bugs in ML systems.
- Lesson 2881 — What is a Feature Store and Why It Matters
- Trajectory analysis
- Trajectory analysis means examining the complete chain of reasoning steps, tool calls, observations, and actions the agent took—its "trajectory"—to understand the failure mode.
- Lesson 2128 — Trajectory Analysis and Error Attribution
- Trajectory Management
- Trajectory Management means tracking the full reasoning chain.
- Lesson 1908 — Implementing ReAct Agents
- Transfer learning
- Adapts parameters through gradient descent on the target task. The adaptation strategy (learning rate, which layers to fine-tune) is chosen manually.
- Lesson 2607 — Meta-Learning vs Transfer Learning
- transferability
- This phenomenon is called transferability, and it's both fascinating and concerning for security.
- Lesson 3381 — Transferability of Adversarial Examples
- Transformation logic
- The actual computation (e.g., aggregations, encodings, time-windowed calculations)
- Lesson 2885 — Feature Definition and Registration
- Translation and structured input-output tasks
- Translation and structured input-output tasks are ideal for encoder-decoder models:
- Lesson 1102 — Encoder-Decoder vs Decoder-Only Trade-offs
- Translation invariance
- The filter detects the same pattern regardless of where it appears in the input
- Lesson 852 — Convolution as a Sliding Window
- Transparency
- Stakeholders should understand how models work, what data they use, and how decisions are made. This includes model cards, documentation, and explainability techniques—especially important when explaining limitations to non-technical users.
- Lesson 3487 — Principles of Responsible AI DevelopmentLesson 3505 — Algorithmic Transparency and Explainability Requirements
- tree
- A tree is a connected graph with no cycles where every node (except the root) has exactly one parent.
- Lesson 2488 — Common Graph Types: Trees, DAGs, and Bipartite Graphs
- Trend detection
- A 30-day moving average reveals medium-term trends better than daily noise
- Lesson 2392 — Rolling Window Statistics
- Truncation Trick
- At inference, BigGAN samples latent codes from a truncated normal distribution (cutting off extreme values). This trades diversity for quality—like choosing to stay closer to "safe" areas of the latent space where the model is most confident.
- Lesson 1489 — BigGAN: Scaling Up GAN Training
- trust region
- A trust region is a constrained neighborhood around your current policy parameters where you "trust" your local linear or quadratic approximation of the objective function.
- Lesson 1791 — The Trust Region ConstraintLesson 2291 — Trust Regions in Optimization
U
- UMAP
- UMAP is more deterministic with default settings, producing more consistent results across runs.
- Lesson 403 — UMAP vs t-SNE: Comparative Analysis
- underfitting
- High bias (also called underfitting) occurs when your model is too simple to learn the underlying patterns in your data.
- Lesson 521 — High Bias Diagnosis
- Unified pretraining and generation
- The same causal attention used during pretraining (next-token prediction) works seamlessly at inference
- Lesson 1200 — Decoder-Only Design: Why GPT Diverged from BERT
- union
- The union is the total area covered by either circle.
- Lesson 947 — Intersection over Union (IoU)
- Units confusion
- SHAP values are in the model's output units (log-odds for classifiers, not probabilities)
- Lesson 3218 — SHAP in Practice: Implementation and Interpretation
- Universal
- A single patch can fool the model on many different images
- Lesson 3385 — Adversarial Patches
- Unsupervised
- No labels at all. Use Isolation Forest, One-Class SVM, or statistical methods. Good when anomalies are truly unknown.
- Lesson 380 — Anomaly Detection in Practice
- Unweighted graphs
- All edges are equal (you're either friends or not)
- Lesson 2483 — What Is a Graph? Nodes, Edges, and Basic Terminology
- update rule
- The update rule is the formula that tells you exactly how to adjust your parameters after each step.
- Lesson 213 — The Gradient Descent Update Rule
- User
- Think of it like a play: the system is the director's notes (invisible to the audience but guiding everything), the user is the audience asking questions, and the assistant is the actor performing on stage.
- Lesson 1232 — Instruction Format and Template DesignLesson 1752 — Instruction Format and TemplatesLesson 1854 — System vs User vs Assistant Messages
- User-based
- The clerk finds other customers with similar purchase histories to yours, then suggests books those customers bought that you haven't.
- Lesson 2350 — User-Based vs Item-Based Approaches
- Utilization rate
- A GPU at 100% utilization drawing full power versus 50% utilization with proportionally less
- Lesson 3469 — GPU Power Consumption and Efficiency
V
- validation curve
- A validation curve is a diagnostic plot that shows how both training and validation scores change as you vary a single hyperparameter.
- Lesson 524 — Validation Curves for Hyperparameters
- value
- Think of it like searching a library: your query is "books about transformers," each book's key is its title and topic tags, and the value is the book's actual content.
- Lesson 1051 — Query, Key, Value: The Three Vectors
- Variance change
- Data that was tightly clustered (std=5) is now highly variable (std=25)
- Lesson 3053 — Statistical Summary Monitoring
- vector database
- A vector database is a specialized storage system optimized for:
- Lesson 1957 — What Is a Vector Database and Why RAG Needs It
- Versioned Test Sets
- The infrastructure maintains multiple test set versions (public validation sets for development, private test sets for final ranking). This prevents overfitting to public benchmarks while still allowing researchers to iterate.
- Lesson 3125 — Leaderboards and Evaluation Infrastructure
- Vertical lines
- Certain words (like punctuation or important keywords) get attention from many positions—these are "hub" words.
- Lesson 1059 — Understanding Attention Weight Visualization
- Vertical scatter
- Wide spread means the feature's impact varies greatly
- Lesson 3213 — SHAP Summary Plots and Feature Importance
- ViLT
- Single-stream architecture where image and text tokens are concatenated early and processed jointly
- Lesson 1412 — Transformer-Based VQA Models
- Visual Genome
- Visual Genome is a landmark dataset that revolutionized this field by providing unprecedented detail about images.
- Lesson 1384 — Visual Genome and Large-Scale VL Datasets
- Visual priming
- Certain objects correlate strongly with specific answers (e.g., "tennis" often appears when rackets are visible).
- Lesson 1413 — VQA Evaluation and Bias Challenges
- Volume explosion
- The "space" becomes so vast that data points are increasingly sparse
- Lesson 1961 — The Curse of Dimensionality in Vector Search
- Vᵀ
- Vᵀ is the transpose of an n×n orthogonal matrix (second rotation)
- Lesson 22 — Singular Value Decomposition (SVD): Concept
W
- Weaviate
- Weaviate is an open-source vector database you deploy yourself (or use their managed cloud).
- Lesson 1966 — Vector Database Options: Pinecone, Weaviate, Qdrant
- weight decay
- That's why L2 regularization is also called weight decay—it makes weights shrink slightly with every training step, unless the original loss function strongly demands they stay large.
- Lesson 734 — L2 Regularization (Weight Decay) Fundamentals
- Weight Dropping
- Weight Dropping is a related technique often used in recurrent networks, where specific weight matrices (like recurrent connections) have dropout applied to them consistently across time steps.
- Lesson 747 — DropConnect and Weight Dropping
- Weight sharing
- The same filter scans the entire image, drastically reducing parameters
- Lesson 889 — LeNet-5: The First Successful CNN
- Weight-only quantization
- Weight-only quantization is a selective approach where you convert model weights (the learned parameters) from 32-bit floating point to lower precision (typically 8-bit integers), but leave activations at full precision during inference.
- Lesson 2633 — Weight-Only Quantization
- Weighted aggregation
- Weighted aggregation is the most common approach: assign importance weights to each metric and compute a composite score:
- Lesson 3101 — Multi-Task and Multi-Objective Evaluation
- Weights
- Weights are static parameters that change slowly through gradient updates.
- Lesson 2648 — QAT for Activations vs Weights
- When unsure
- The memory saved is often negligible compared to the risk of gradient errors
- Lesson 786 — In-place Operations and Memory
- who
- Having risk management processes isn't enough—you need a governance structure that defines who is responsible for what, how decisions get made, and where accountability lies when things go wrong.
- Lesson 3536 — Risk Governance Structures
- Why it matters
- A non-trivial null space means the transformation loses information. Multiple different inputs produce the same output (or zero), making the transformation non-invertible.
- Lesson 12 — Column Space and Null Space
- Why this works
- The power law relationship continues beyond the compute-optimal point—just more slowly. You get better downstream task performance, stronger reasoning, and improved generalization.
- Lesson 1630 — Post-Chinchilla Training Strategies
- wide component
- The wide component is a linear model that learns direct feature interactions—essentially memorizing co-occurrence patterns through cross-product transformations.
- Lesson 2367 — Wide & Deep Networks for Recommendations
- Width
- Width means the hidden dimension size (embedding dimension):
- Lesson 596 — Network Architecture Terminology: Depth and WidthLesson 600 — Depth vs Width: Architectural Trade-offsLesson 1349 — ViT Model Variants
- Win rate
- Win rate is the simplest preference metric: given two model outputs for the same input, a human (or LLM judge) picks the better one.
- Lesson 3173 — Introduction to Win Rate Metrics
- Word embeddings
- Word embeddings are dense, low-dimensional vectors (typically 50-300 dimensions) where similar words have similar vectors.
- Lesson 1117 — Why Word Embeddings: From One-Hot to Dense Vectors
- WordPiece
- WordPiece is more selective—it merges pairs that maximize likelihood, creating a vocabulary that better reflects language patterns rather than raw frequency.
- Lesson 1264 — Comparing Tokenization Algorithms
- Worker count increases
- More participants in the All-Reduce means more coordination complexity
- Lesson 2711 — Communication Overhead and Bottlenecks
- world size
- The rank is a unique ID for each process (0, 1, 2, ...), the world size is the total number of processes, and a process group is the communication channel connecting them all.
- Lesson 2794 — Distributed Process Groups and Ranks
X
- X^T
- X^T is the transpose of your feature matrix
- Lesson 193 — The Closed-Form Solution (Normal Equation)
Z
- Zero-copy operations
- Branches share underlying data objects; only changes are stored separately.
- Lesson 2844 — LakeFS for Data Lake Versioning
- Zero-Shot Chain-of-Thought
- Zero-Shot Chain-of-Thought is remarkably simple: just append the phrase "Let's think step by step" (or similar variants) to your prompt.
- Lesson 1864 — Zero-Shot Chain-of-Thought with 'Let's Think Step by Step'
- Zero-shot forecasting
- Zero-shot forecasting means you can feed your time series directly into a pre-trained model like TimeGPT and get predictions immediately—no task-specific training required.
- Lesson 2425 — Zero-Shot Forecasting with Foundation Models
- zero-shot learning
- This is called zero-shot learning: the model performs a task with zero training examples for that specific task.
- Lesson 1203 — GPT-2's Zero-Shot Task Transfer
- Zero-shot QA
- Zero-shot QA means giving the model a question with context and expecting an answer—no examples provided.
- Lesson 1310 — QA with Large Language Models