Course contentsShow
← AI Engineering
Lesson 233 of 1,8866. Embeddings and Semantic SearchFree lesson

Query Preprocessing and Normalization

Techniques for cleaning and normalizing queries before encoding to improve search quality.

Query Preprocessing and Normalization

What you'll learn: How to clean and standardize user queries before embedding them to improve semantic search accuracy and consistency.

Why Preprocess Queries?

When users search for information, their queries can be messy: extra spaces, inconsistent capitalization, typos, or unnecessary punctuation. If you embed a query like " WHAT is AI?!? " directly, you'll get a different vector than "what is ai" — even though they mean the same thing. This inconsistency can hurt search quality.

Think of it like preparing ingredients before cooking. You wash vegetables, trim excess parts, and cut them uniformly so they cook evenly. Similarly, preprocessing ensures your query embeddings are clean and comparable.

Common Preprocessing Techniques

Lowercasing: Convert all text to lowercase for consistency.

query = query.lower()

Trimming Whitespace: Remove leading/trailing spaces and collapse multiple spaces.

query = " ".join(query.split())

Removing Special Characters: Strip punctuation that doesn't add semantic value.

import string
query = query.translate(str.maketrans('', '', string.punctuation))

Basic Typo Correction: While advanced spell-checking isn't always necessary, catching common errors can help.

When to Apply Normalization

Apply the same preprocessing to both:

  1. Document text when building your embedding index
  2. User queries at search time

This ensures queries and documents exist in the same "cleaned" vector space, improving similarity matching.

Practical Example

def preprocess_query(query):
    query = query.lower()
    query = " ".join(query.split())
    query = query.translate(str.maketrans('', '', string.punctuation))
    return query

raw_query = "  What's the BEST AI model?!  "
clean_query = preprocess_query(raw_query)
# Result: "whats the best ai model"

Key Takeaway: Preprocessing queries before embedding ensures consistent vector representations, improving semantic search accuracy by treating semantically identical queries the same way.