Course contentsShow
← Machine Learning and Deep Learning
Lesson 2920 of 3,53863. Model Serving and Inference InfrastructureFree lesson

Cache Key Design and Hashing

Designing effective cache keys for model inputs, including handling numerical precision and input normalization.

Cache Key Design and Hashing

You'll learn how to design effective cache keys for model inputs to maximize cache hit rates while avoiding collisions and handling numerical edge cases.

The Core Challenge

When caching inference results (as introduced in the previous lesson), you need a reliable way to identify "identical" inputs. Your cache key must uniquely represent each input so that when the same input appears again, you retrieve the cached result instead of re-running inference.

The challenge? Raw model inputs are often complex: floating-point tensors, variable-length sequences, or nested dictionaries. You need to transform these into consistent, hashable identifiers.

Key Design Principles

Normalization Before Hashing

Two inputs that are semantically identical might differ in representation. Normalize before generating keys:

  • Numerical precision: Round floats to appropriate precision (e.g., 6 decimals) to treat 0.123456789 and 0.123456712 as identical
  • Input scaling: Apply the same preprocessing pipeline used during training
  • Whitespace/case: Normalize text inputs (strip, lowercase)

Collision-Resistant Hashing

Use cryptographic hash functions like SHA-256 or fast non-cryptographic ones like xxHash for production systems. Simple string concatenation risks collisions.

Deterministic Serialization

Arrays and objects must serialize consistently. json.dumps(obj, sort_keys=True) ensures dictionaries with identical content produce identical strings regardless of insertion order.

Practical Considerations

For high-dimensional inputs like images, consider perceptual hashing or feature extraction rather than raw pixel values—this allows similar (not just identical) inputs to share cache entries.

Balance key complexity with computational cost: hashing a 4K image shouldn't take longer than inference itself.

Key Takeaway: Effective cache keys require normalized, deterministic input representation combined with fast, collision-resistant hashing—treating floating-point precision and input preprocessing as integral parts of the key generation strategy.