Creating and Managing Snapshots
What You'll Learn
You'll learn how to implement snapshot tests for AI systems using pytest-snapshot, maintain baseline outputs, and keep snapshots up-to-date as your system evolves.
The Core Idea
Snapshot testing captures your system's actual output and saves it as a reference file. Future test runs compare new outputs against this saved "snapshot." If outputs match, the test passes. If they differ, you either found a regression or need to update your baseline.
Think of it like taking a photograph of your AI system's response. Every time you run the test, you compare the new image to the original photo. Changed unintentionally? That's a bug. Changed because you improved something? Update the photo.
Implementing with pytest-snapshot
# test_rag_outputs.py
def test_retrieval_response(snapshot):
query = "What is machine learning?"
result = rag_pipeline.query(query)
# First run creates snapshots/__init___test_retrieval_response.txt
snapshot.assert_match(result, "retrieval_response.txt")
def test_parsed_output(snapshot):
raw_output = llm.generate("List three AI applications")
parsed = output_parser.parse(raw_output)
# Snapshot structured data as JSON
snapshot.assert_match(parsed, "parsed_output.json")
Managing Your Snapshots
When outputs legitimately change (new prompt, better model, refined retrieval), update snapshots:
pytest --snapshot-update
Keep snapshots in version control alongside tests. Review snapshot diffs during code review—they show exactly how system behavior changed.
Key Takeaway: Snapshot testing provides regression detection for AI outputs by comparing against saved baselines; update snapshots deliberately when behavior changes are intentional, and always review the diffs to ensure changes align with expectations.