In this lesson, you will learn how to shrink your storage footprint by using Run-Length Encoding (RLE) to represent long sequences of identical values in low-cardinality columns.
The Concept of RLE
When we store data in columnar formats, we often encounter columns with "low cardinality"—meaning they contain a small number of unique values that repeat frequently, such as a status column (e.g., "PENDING", "SUCCESS", "FAILED") or a boolean flag.
Storing the raw string "PENDING" a million times is wasteful. Run-Length Encoding (RLE) optimizes this by storing a value followed by the number of times it repeats (its "run"). Instead of a long list of redundant entries, we store the value and its count, drastically reducing the bytes required on disk.
A Concrete Illustration
Imagine a column containing the status of sensor events:
['ACTIVE', 'ACTIVE', 'ACTIVE', 'INACTIVE', 'INACTIVE']
Without compression, this takes five storage slots. With RLE, we transform this into a pair-based format:
('ACTIVE', 3)('INACTIVE', 2)
This shrinks the data significantly. During query execution, the database engine reads these pairs and expands them back into the original stream on the fly. Because modern CPUs are excellent at performing these simple expansions, the decompression cost is often lower than the time saved by reading fewer bytes from the disk.
Implementation Logic
If you were to implement a basic RLE encoder, the logic looks like this:
# Simple conceptual representation
data = ["A", "A", "A", "B", "B", "C"]
# RLE result: [("A", 3), ("B", 2), ("C", 1)]
When dealing with large datasets in columnar storage, RLE is most effective when the data is sorted. If your "ACTIVE" statuses are scattered throughout the column, RLE cannot create long runs and the compression ratio suffers. By sorting the data, you group identical values together, creating the long runs that RLE needs to maximize efficiency.
Key Takeaway: Run-Length Encoding reduces storage overhead for low-cardinality data by replacing repeating sequences with (value, count) pairs, significantly optimizing disk I/O when combined with sorted column layouts.