Leveraging Bit-Packing for Integer Compression

Apply bit-packing techniques to minimize the storage size of integer data by packing values into the minimum required number of bits.

In this lesson, you will learn how to shrink your storage footprint by stripping away unnecessary empty bits from your integer data using bit-packing.

The Problem: Wasted Space

In standard computer systems, integers are often stored in fixed 32-bit or 64-bit containers. However, if you are storing a list of values like age (0–120) or department IDs (1–50), you don’t need 32 bits for each number. An age of 120 only requires 7 bits ( $2^7 = 128$ ). If you store that 7-bit value in a 32-bit integer, you are wasting 25 bits per record. Over millions of rows, that overhead becomes massive.

The Solution: Bit-Packing

Bit-packing is the process of discarding those "wasted" high-order bits and packing the significant bits of multiple integers tightly together into a single word.

Imagine you have three integers that only require 3 bits each. Instead of using three 32-bit slots (96 bits total), you can pack them side-by-side into a single 9-bit sequence:

Value A (3 bits) | Value B (3 bits) | Value C (3 bits)
[ 1 0 1 ]        [ 0 1 1 ]          [ 1 1 0 ]

By shifting the bits and using bitwise masks, you can pack these into a single 64-bit block. When your system reads the data back, it performs the inverse operation—masking and shifting—to extract the individual numbers.

Implementation Logic

When we "pack" these values, we effectively remove the padding, creating a dense stream of bits.

# Conceptual packing of two 3-bit values into one 6-bit byte
val1 = 5  # Binary: 101
val2 = 3  # Binary: 011

# Pack by shifting val1 to the left and adding val2
packed = (val1 << 3) | val2 
# Result: 101011 (binary) which is 43 in decimal

This technique is incredibly fast for modern CPUs, which are optimized for these logical operations. Because you are reading significantly fewer bytes from disk or memory, you reduce I/O overhead and increase the efficiency of your CPU cache, leading to much faster analytical query performance.

Key Takeaway: Bit-packing reduces storage requirements by identifying the minimum bit-width necessary to represent a range of integers and packing them tightly to eliminate redundant padding bits.