Measuring Prompt Token Usage
What you'll learn: How to accurately count tokens in your prompts using tokenizers and pinpoint which parts consume the most tokens.
Why Counting Tokens Accurately Matters
In the previous lesson, you learned why prompt length matters—it affects latency, cost, and context window capacity. But how do you actually measure it? You can't just count words or characters. LLMs don't process text that way; they process tokens, and a token might be a whole word, part of a word, or even punctuation.
Think of it like measuring ingredients for a recipe. If you eyeball "a cup of flour," you might be off by 20%. But if you use a scale and measure grams, you get precision. Tokenizers are your scale for prompts.
Using Tokenizers to Count Tokens
Every model family (GPT, Claude, Llama, etc.) uses its own tokenizer. The same text can produce different token counts across models. To measure accurately:
- Use the exact tokenizer for your model (e.g.,
tiktokenfor OpenAI models, the model's Hugging Face tokenizer for others) - Encode your full prompt including system messages, few-shot examples, and user input
- Count tokens per component to identify what's consuming your budget
For example, you might discover that your 5 few-shot examples consume 800 tokens while your actual user query is only 50 tokens. That's actionable data—you now know where to optimize.
Identifying the Largest Components
Break your prompt into logical pieces (system prompt, examples, context, instructions, user input) and measure each separately. This reveals optimization opportunities: maybe you can compress verbose instructions or reduce example count without sacrificing quality.
Key Takeaway: Always measure prompt tokens using the model's specific tokenizer, not character counts, and profile individual prompt components to find optimization targets.