Handling Schema Evolution in Avro

In this lesson, you will learn how to use Avro schemas to evolve your data structures over time without breaking the systems that depend on your files.

The Problem: Data Over Time

In a production data pipeline, your data structure is rarely static. You might add a new field (e.g., user_phone) or remove an obsolete one. If you were using simple formats like CSV, you would have to rewrite your entire historical dataset every time you added a column to avoid breaking downstream consumers. Avro solves this by storing the schema inside the file alongside the binary data.

Schema Resolution

Avro uses a concept called "Schema Resolution." When a reader processes an Avro file, it compares the Writer’s Schema (the schema used to write the data) with the Reader’s Schema (the schema the application expects).

Avro uses these rules to bridge the gap:

1. Adding a field: If you add a field, you must provide a default value in the schema. When an old file is read, Avro automatically injects that default value for the missing field.
2. Removing a field: If a field exists in the file but is missing from the reader's schema, Avro simply ignores it.

A Practical Example

Imagine your original schema for a user profile:

{
  "type": "record",
  "name": "User",
  "fields": [{"name": "id", "type": "int"}]
}

Later, you want to add an email address. You update your schema, ensuring you include a default:

{
  "type": "record",
  "name": "User",
  "fields": [
    {"name": "id", "type": "int"},
    {"name": "email", "type": "string", "default": "n/a"}
  ]
}

Because you included the default value, an application using this new schema can successfully read old files that only contain the id. The Avro library detects the mismatch, sees the default value, and silently fills in "n/a" for the email field of those older records. This allows you to update your storage format continuously without ever needing to perform a "migration" or rewrite your back-catalog of files.

Key Takeaway: By storing the schema with the data and enforcing default values for new fields, Avro allows downstream consumers to read evolving data formats without breaking.