Bits-Per-Character and Bits-Per-Byte Metrics
What you'll learn: How to measure language model compression efficiency at the character and byte level, offering finer-grained alternatives to word-level perplexity.
The Core Idea
While perplexity evaluates models based on predicting whole tokens (often words or subwords), bits-per-character (BPC) and bits-per-byte (BPB) measure how many bits of information are needed to encode each individual character or byte in a sequence. These are compression-based metrics that answer: "How efficiently can the model compress the text?"
Think of it like comparing file compression algorithms. A better compressor needs fewer bits to represent the same information. Similarly, a better language model assigns higher probabilities to actual sequences, requiring fewer bits to encode them.
How They Work
Both metrics derive from cross-entropy loss at the character or byte level:
- BPC = cross-entropy loss / log(2) computed per character
- BPB = cross-entropy loss / log(2) computed per byte
The division by log(2) converts from natural log units (nats) to bits. Lower values indicate better models—the model is "surprised" less by the actual data.
Why Use These Metrics?
Language-agnostic evaluation: Character and byte-level metrics work across any writing system without requiring language-specific tokenization. This makes them ideal for multilingual models or languages with complex morphology.
Tokenization-independent: Unlike perplexity, which depends on your tokenizer's vocabulary, BPC and BPB provide consistent comparisons even when models use different tokenization schemes.
Fine-grained analysis: These metrics capture model quality on smaller units, revealing how well models handle character patterns, spelling, and low-level structure.
Key Takeaway: Bits-per-character and bits-per-byte metrics evaluate language models through a compression lens, measuring efficiency at character or byte granularity—offering tokenization-independent, language-agnostic alternatives to traditional perplexity.