Token Counting Basics
What you'll learn: How to accurately count tokens using tokenizers and understand the difference between input and output tokens for cost optimization.
Why Exact Token Counts Matter
You already know why token usage matters in production. Now let's get precise. Token counting isn't as simple as counting words—LLMs use tokenizers that split text into subword units. The word "tokenization" might be 3 tokens, while "AI" could be 1 or 2, depending on the model.
Without accurate counting, your cost estimates will be wrong, your budgets will fail, and you'll either over-provision or hit unexpected rate limits.
The Two Types of Tokens
Every API call has two distinct token counts:
- Input tokens (prompt tokens): Everything you send to the model—system messages, user prompts, examples, context
- Output tokens (completion tokens): Everything the model generates in response
These are typically priced differently. Output tokens often cost 2-3× more than input tokens because generation is computationally expensive.
Using Tokenizers
Think of a tokenizer as a measuring tape for text. Just like you wouldn't estimate lumber by eye, don't estimate tokens by word count.
For OpenAI models, use tiktoken. For others, use the model's specific tokenizer from Hugging Face or the provider's SDK. Always use the exact tokenizer that matches your target model—GPT-4 and GPT-3.5 use different tokenizers.
Count before you send requests to predict costs and avoid surprises. Count after responses arrive to validate billing and track actual usage.
Input vs Output in Practice
If you send a 500-token prompt and receive a 150-token response, you're charged for 500 input + 150 output tokens. Your cost calculation must treat these separately based on the provider's pricing tier.
Key Takeaway: Use the exact tokenizer for your model to count input and output tokens separately—this is the foundation of accurate cost tracking and optimization.