Course contentsShow
← AI Engineering
Lesson 1176 of 1,88629. Cost OptimizationFree lesson

Token Counting Basics

How to count tokens accurately using tokenizers and understanding the difference between input and output tokens.

Token Counting Basics

What you'll learn: How to accurately count tokens using tokenizers and understand the difference between input and output tokens for cost optimization.

Why Exact Token Counts Matter

You already know why token usage matters in production. Now let's get precise. Token counting isn't as simple as counting words—LLMs use tokenizers that split text into subword units. The word "tokenization" might be 3 tokens, while "AI" could be 1 or 2, depending on the model.

Without accurate counting, your cost estimates will be wrong, your budgets will fail, and you'll either over-provision or hit unexpected rate limits.

The Two Types of Tokens

Every API call has two distinct token counts:

  • Input tokens (prompt tokens): Everything you send to the model—system messages, user prompts, examples, context
  • Output tokens (completion tokens): Everything the model generates in response

These are typically priced differently. Output tokens often cost 2-3× more than input tokens because generation is computationally expensive.

Using Tokenizers

Think of a tokenizer as a measuring tape for text. Just like you wouldn't estimate lumber by eye, don't estimate tokens by word count.

For OpenAI models, use tiktoken. For others, use the model's specific tokenizer from Hugging Face or the provider's SDK. Always use the exact tokenizer that matches your target model—GPT-4 and GPT-3.5 use different tokenizers.

Count before you send requests to predict costs and avoid surprises. Count after responses arrive to validate billing and track actual usage.

Input vs Output in Practice

If you send a 500-token prompt and receive a 150-token response, you're charged for 500 input + 150 output tokens. Your cost calculation must treat these separately based on the provider's pricing tier.

Key Takeaway: Use the exact tokenizer for your model to count input and output tokens separately—this is the foundation of accurate cost tracking and optimization.