Sources of Bias: Model Architecture and Objectives
You'll learn how the fundamental design decisions you make when building AI models—architecture choices, loss functions, and optimization goals—can bake in systematic biases before training even begins.
The Architecture Isn't Neutral
When you design or select a model architecture, you're making assumptions about what patterns matter. A convolutional neural network assumes spatial locality is important—great for images, but this "inductive bias" might miss global context. Transformer architectures prioritize attention patterns that can inadvertently amplify whatever correlations appear most frequently in training data, even harmful ones.
Think of it like building a house: if you design it with only right-angle walls, you've constrained the shapes of rooms you can create. Similarly, architectural choices like layer depth, attention mechanisms, or tokenization schemes constrain what your model can and cannot learn.
Objectives Shape Behavior
Your loss function is your model's north star—it optimizes ruthlessly for whatever you measure. If you optimize purely for accuracy on a dataset where one demographic is overrepresented, the model learns to excel at predictions for that group while underperforming on others.
Consider a language model trained with next-token prediction: it learns to reproduce statistical patterns, including stereotypes. If technical documentation in your corpus consistently uses "he" for engineers, that objective function rewards perpetuating that pattern.
The Optimization Process
Even how you optimize matters. Gradient descent can get stuck in local optima that favor majority patterns. Regularization techniques might inadvertently suppress features important to minority groups if those features appear "noisy" statistically.
Key Takeaway: Bias enters your AI system through the very structure you choose, the goals you set, and how you optimize—these design decisions encode assumptions about what matters before a single data point is processed.