Tokenization and Context Window
What you'll learn: How tokenizer efficiency directly affects how much real content you can fit into a model's fixed context window.
The Compression Challenge
Every LLM has a fixed context window — a maximum number of tokens it can process at once (e.g., 2048, 4096, or 8192 tokens). But here's the catch: different content compresses into tokens very differently depending on your tokenizer.
Think of it like packing a suitcase. You have a fixed size (context window), but some items (languages, content types) pack more efficiently than others.
Language and Content Variation
English text typically compresses well because BPE tokenizers are often trained heavily on English data. A word might become 1-2 tokens.
Non-Latin scripts (Chinese, Arabic, Korean) often compress poorly — a single character might require multiple tokens. This means a Chinese document uses up your context window much faster than an equivalent English one.
Code compression varies by language. Python with common keywords tokenizes efficiently, but dense technical code with unusual variable names fragments into many tokens.
Numbers and special characters are particularly inefficient — long numbers might tokenize as individual digits, wasting precious context slots.
Real Impact
Two documents with identical character counts can have vastly different token counts. A 2000-character English article might use 500 tokens, while a 2000-character Chinese article could use 1500 tokens — meaning you fit 3× less content in the same context window.
This creates real disparities: your model can "see" more English context than multilingual context, affecting performance across languages.
Key Takeaway: Tokenizer compression ratios aren't uniform — they vary dramatically by language and content type, making your effective context window much smaller for some languages and content than others, even though the token limit stays fixed.