Context Window Constraints
What you'll learn: How token limits restrict how much conversation history you can include in each API call, and why managing this "memory" is critical for chatbots.
The Token Budget Problem
Every LLM has a context window — a maximum number of tokens it can process in a single request. This includes your system prompt, all message history, and the current user input combined.
Think of it like a whiteboard: you can only fit so much writing on it. If you want to add new information, you might need to erase something old.
How Conversation History Consumes Space
Each message in your conversation history takes up tokens:
- User message: "What's the weather?" (~4 tokens)
- Assistant response: "I don't have access to real-time weather..." (~15 tokens)
- Next user message: "Can you check?" (~4 tokens)
As conversations grow, your history accumulates. A 50-message conversation could easily consume 2,000–5,000 tokens before you even send the next query.
Model-Specific Limits
Different models have different capacities:
- GPT-3.5-turbo: 4,096 or 16,385 tokens
- GPT-4: 8,192 or 32,768 tokens
- Claude 3: Up to 200,000 tokens
- Llama models: Varies (often 4,096–8,192)
If you exceed the limit, the API will reject your request or truncate content unexpectedly.
Why This Matters for Chatbots
Without careful management, long conversations fail. You must decide: keep recent messages? Summarize older ones? Drop the middle? These pruning strategies (covered in earlier lessons) directly respond to context window constraints.
Key Takeaway: Context windows are finite; every token of conversation history counts against your model's limit, requiring strategic message retention and pruning in production chatbots.