Low-Level Serialization Layouts

In this lesson, you will learn how to manually arrange bits within a byte array to create high-performance, wire-ready data structures without relying on heavy serialization frameworks.

The Problem with High-Level Structs

When we define a struct in Rust, the compiler inserts padding between fields to satisfy CPU alignment requirements. While this is great for performance during runtime, it is inefficient for wire protocols where bandwidth is expensive. If you send a standard struct over a network, you are often transmitting "dead" padding bytes that carry no information.

Manual Bit-Packing

To minimize our payload, we perform manual bit-packing. Instead of using native types like u32, we map our data to the smallest possible bit-width and use bitwise operations to pack multiple fields into a single byte range.

Consider a protocol requiring a status (3 bits), priority (2 bits), and flag (1 bit) that we want to cram into a single byte:

#[repr(C)]
struct Packet {
    data: u8,
}

impl Packet {
    // Layout: [Status: 3][Priority: 2][Flag: 1][Reserved: 2]
    fn new(status: u8, priority: u8, flag: bool) -> Self {
        let mut data = (status & 0x07) << 5;
        data |= (priority & 0x03) << 3;
        if flag { data |= 1 << 2; }
        Self { data }
    }

    fn status(&self) -> u8 { (self.data >> 5) & 0x07 }
}

Why Manual Layout Matters

By manually calculating offsets, we achieve two things:

1. Determinism: We control exactly which bit represents which value, ensuring cross-platform compatibility regardless of the host's native endianness or padding rules.
2. Density: We eliminate unused space entirely, allowing us to transmit more logical units per MTU (Maximum Transmission Unit) compared to standard serialized formats like JSON or even packed structs that align to byte boundaries.

When implementing this, always ensure your masks and shifts are strictly defined, and use u8 arrays for the buffer to prevent the compiler from reordering or padding your fields, effectively bypassing the standard memory layout rules in favor of your custom wire specification.

Key Takeaway: Manual bit-packing allows you to bypass Rust's memory alignment rules to create zero-waste wire protocols, transforming complex data into a highly dense bit-stream.