Course contentsShow
← Computer Science
Lesson 2836 of 2,87263. Classical ML AlgorithmsFree lesson

Feature Engineering for Logistic Regression

Transform and combine features, handle categorical variables, and scale inputs for better results.

Feature Engineering for Logistic Regression

What you'll learn: How to transform and prepare features to maximize your logistic regression model's performance.

The Core Idea

Logistic regression learns linear decision boundaries in feature space. But real-world patterns are rarely linear in their raw form. Feature engineering bridges this gap by creating new representations that make non-linear patterns accessible to your linear classifier.

Think of it like preparing ingredients before cooking: raw vegetables (raw features) become more useful when chopped, combined, or seasoned (engineered).

Key Transformations

Polynomial Features: Create interactions like x1 * x2 or powers like x1² to capture non-linear relationships. A linear boundary in this enriched space becomes a curve in the original space.

Categorical Variables: Convert categories to numbers using one-hot encoding. For a "color" feature with values {red, blue, green}, create three binary features: is_red, is_blue, is_green. Never use arbitrary numbers (1,2,3) as they imply false ordering.

Feature Scaling: Normalize or standardize numerical features so they're on similar scales. Since gradient descent updates all parameters using the same learning rate, features with vastly different ranges (age: 0-100 vs income: 0-1,000,000) cause slow, erratic convergence. Common approaches: min-max scaling to [0,1] or z-score normalization (mean=0, std=1).

Domain-Specific Features: Combine existing features based on problem knowledge. For predicting loan default, debt_to_income_ratio = debt / income might be more predictive than the raw values alone.

Why It Matters

Well-engineered features help logistic regression find better decision boundaries, improve convergence during training, and often boost accuracy more than algorithm tuning alone.

Key Takeaway: Feature engineering transforms raw data into representations that align with logistic regression's linear nature—through polynomial expansions for non-linearity, proper encoding for categorical data, and scaling for stable optimization.