Course contentsShow
← AI Engineering
Lesson 896 of 1,88622. Testing and CI/CD for AI SystemsFree lesson

Creating and Managing Snapshots

Implementing snapshot tests with tools like pytest-snapshot, managing baseline outputs, and updating snapshots.

Creating and Managing Snapshots

What You'll Learn

You'll learn how to implement snapshot tests for AI systems using pytest-snapshot, maintain baseline outputs, and keep snapshots up-to-date as your system evolves.

The Core Idea

Snapshot testing captures your system's actual output and saves it as a reference file. Future test runs compare new outputs against this saved "snapshot." If outputs match, the test passes. If they differ, you either found a regression or need to update your baseline.

Think of it like taking a photograph of your AI system's response. Every time you run the test, you compare the new image to the original photo. Changed unintentionally? That's a bug. Changed because you improved something? Update the photo.

Implementing with pytest-snapshot

# test_rag_outputs.py
def test_retrieval_response(snapshot):
    query = "What is machine learning?"
    result = rag_pipeline.query(query)
    
    # First run creates snapshots/__init___test_retrieval_response.txt
    snapshot.assert_match(result, "retrieval_response.txt")

def test_parsed_output(snapshot):
    raw_output = llm.generate("List three AI applications")
    parsed = output_parser.parse(raw_output)
    
    # Snapshot structured data as JSON
    snapshot.assert_match(parsed, "parsed_output.json")

Managing Your Snapshots

When outputs legitimately change (new prompt, better model, refined retrieval), update snapshots:

pytest --snapshot-update

Keep snapshots in version control alongside tests. Review snapshot diffs during code review—they show exactly how system behavior changed.

Key Takeaway: Snapshot testing provides regression detection for AI outputs by comparing against saved baselines; update snapshots deliberately when behavior changes are intentional, and always review the diffs to ensure changes align with expectations.