Course contentsShow
← Machine Learning and Deep Learning
Lesson 426 of 3,53810. Feature EngineeringFree lesson

Handling Unseen Categories at Test Time

Strategies for dealing with new category values that appear in production but weren't in training data.

Handling Unseen Categories at Test Time

What you'll learn: How to gracefully handle new categorical values that appear in production data but weren't present during model training.

The Problem

Imagine you trained a model on customer data with cities: "New York," "London," and "Tokyo." Your model learned encodings for these three cities. Now in production, a customer from "Paris" appears—a city your model has never seen. What happens?

Without a strategy, your model will likely crash or produce errors because "Paris" doesn't exist in its encoding mapping.

Common Strategies

1. Unknown Category Placeholder

Create a catch-all category like "OTHER" or "UNKNOWN" during training. Map all unseen categories to this placeholder at test time.

# During encoding setup
encoder.handle_unknown = 'use_encoded_value'
encoder.unknown_value = -1  # or map to "OTHER" category

2. Ignore/Drop Strategy

Simply treat unseen categories as missing values. This works when the category isn't critical to predictions, but you lose information.

3. Default Value Assignment

For target encoding or frequency encoding, assign the global mean or a conservative default value to new categories.

4. Retrain Regularly

In production systems, periodically retrain your model with updated data that includes newly emerged categories.

Why This Matters

Real-world data is messy and evolving. New product categories emerge, customers move to new cities, and businesses expand. Your encoding strategy must be robust to change, or your deployed model becomes brittle and unreliable.

Key Takeaway: Always configure your categorical encoders to handle unknown values gracefully—either by mapping to a special "UNKNOWN" category, assigning default values, or treating them as missing data. Never assume test data will perfectly match training data.