In this lesson, you will learn how to shrink your storage footprint by using bit-packing to store integers using only the exact number of bits they require.
The Problem with Fixed-Width Storage
In most systems, we store integers using fixed-width data types, such as 32-bit or 64-bit integers. If you have a column containing values ranging from 0 to 7 (like a "Day of the Week" index), a 32-bit integer is incredibly wasteful. It uses 32 bits to store information that only needs 3 bits ( possible values). When you store millions of these rows, that wasted space compounds, leading to higher I/O overhead and increased disk usage.
How Bit-Packing Works
Bit-packing is the technique of discarding the unused "padding" bits and packing raw values tightly together. Instead of aligning each number to a byte boundary (8, 16, or 32 bits), we treat the data as a continuous stream of bits.
Imagine you have three values: 5, 2, and 7. In binary, these are 101, 010, and 111.
- Standard Storage (32-bit integers): bits.
- Bit-Packed (3-bit width):
101+010+111= 9 bits.
By stripping away the 29 leading zeros from each number, we’ve compressed our storage by over 90%.
Implementation Concept
When we write these values, we perform bitwise operations to shift the integers into position within a larger buffer. Here is a simplified mental model of how the bits pack together:
# Conceptual packing of three 3-bit values into a single byte
val1 = 5 # 101
val2 = 2 # 010
val3 = 7 # 111
# The packed result (9 bits total): 101010111
# We store this across two bytes rather than three 32-bit integers.
When reading this data back, the system reads the specific bit-width defined in the metadata, shifts the bits to isolate the value, and masks out the remaining junk data. This is highly efficient for analytical queries where we perform column projections, as the CPU can process these packed streams very quickly using low-level bitwise instructions.
Key Takeaway: Bit-packing reduces storage costs and improves I/O performance by eliminating unused bits, allowing you to fit more data into the same amount of memory and disk space.