Course contentsShow
← System Design
Lesson 725 of 1,91919. Event Streaming PlatformsFree lesson

Schema Registry and Evolution

Managing message schemas centrally with compatibility checks to enable safe schema evolution across producers and consumers.

Schema Registry and Evolution

What You'll Learn

You'll learn how a centralized schema registry prevents data chaos by managing message formats and ensuring producers and consumers can safely evolve without breaking each other.

The Problem: Schema Chaos

Imagine a Kafka topic where producers send user events. Without coordination, Producer A might send {"userId": 123, "name": "Alice"} while Producer B sends {"user_id": "123", "fullName": "Alice"}. Consumers break because they don't know which format to expect. As systems evolve, this gets worse.

What is a Schema Registry?

A schema registry is a centralized service that stores and validates message schemas (data formats). Before publishing, producers register their schema. Consumers fetch schemas to deserialize messages correctly. Think of it as a "contract enforcement office" for your data.

Popular implementations include Confluent Schema Registry (for Kafka) and cloud-native equivalents like AWS Glue Schema Registry.

How It Works

  1. Producer registers schema: Before sending data, the producer submits the schema (e.g., Avro, Protobuf, JSON Schema) to the registry
  2. Registry validates compatibility: The registry checks if the new schema is compatible with existing versions using rules (backward, forward, full compatibility)
  3. Schema ID embedding: Messages include a small schema ID instead of the full schema, saving bandwidth
  4. Consumer retrieves schema: Consumers fetch the schema by ID to deserialize messages

Schema Evolution and Compatibility

Evolution means changing schemas over time safely:

  • Backward compatible: New consumers can read old messages (add optional fields)
  • Forward compatible: Old consumers can read new messages (remove optional fields)
  • Full compatible: Both directions work (add/remove only optional fields)

The registry blocks incompatible changes, preventing silent breakage.

Key Takeaway:
A schema registry acts as a central contract manager that enforces compatibility rules, enabling safe schema evolution while preventing producers and consumers from breaking each other as data formats change over time.