Cache Key Design and Hashing
You'll learn how to design effective cache keys for model inputs to maximize cache hit rates while avoiding collisions and handling numerical edge cases.
The Core Challenge
When caching inference results (as introduced in the previous lesson), you need a reliable way to identify "identical" inputs. Your cache key must uniquely represent each input so that when the same input appears again, you retrieve the cached result instead of re-running inference.
The challenge? Raw model inputs are often complex: floating-point tensors, variable-length sequences, or nested dictionaries. You need to transform these into consistent, hashable identifiers.
Key Design Principles
Normalization Before Hashing
Two inputs that are semantically identical might differ in representation. Normalize before generating keys:
- Numerical precision: Round floats to appropriate precision (e.g., 6 decimals) to treat
0.123456789and0.123456712as identical - Input scaling: Apply the same preprocessing pipeline used during training
- Whitespace/case: Normalize text inputs (strip, lowercase)
Collision-Resistant Hashing
Use cryptographic hash functions like SHA-256 or fast non-cryptographic ones like xxHash for production systems. Simple string concatenation risks collisions.
Deterministic Serialization
Arrays and objects must serialize consistently. json.dumps(obj, sort_keys=True) ensures dictionaries with identical content produce identical strings regardless of insertion order.
Practical Considerations
For high-dimensional inputs like images, consider perceptual hashing or feature extraction rather than raw pixel values—this allows similar (not just identical) inputs to share cache entries.
Balance key complexity with computational cost: hashing a 4K image shouldn't take longer than inference itself.
Key Takeaway: Effective cache keys require normalized, deterministic input representation combined with fast, collision-resistant hashing—treating floating-point precision and input preprocessing as integral parts of the key generation strategy.