Time-to-First-Token vs Total Generation Time
What you'll learn: How to distinguish between the user's perceived responsiveness and the actual completion time of LLM responses.
When an LLM generates text, two critical latency metrics matter for user experience:
Time-to-First-Token (TTFT)
This measures how long from your request until the first token appears. It includes:
- Prompt processing time
- Model inference startup
- Network overhead
TTFT determines how quickly users see something happening. Even if the full response takes 10 seconds, seeing the first word in 200ms makes the system feel responsive.
Total Generation Time
This measures from request to complete response. It includes:
- TTFT
- Plus all subsequent token generation
- Streaming duration
This is your actual end-to-end latency, but matters less for perceived speed.
Why They Impact UX Differently
Think of it like streaming video: TTFT is like how fast the video starts playing (buffering time), while total generation time is the entire video duration. Users tolerate long videos if they start instantly, but hate short videos that buffer forever.
For chatbots and assistants:
- TTFT < 300ms feels instant and responsive
- TTFT > 2 seconds feels broken, even if total time is reasonable
- Total time matters for throughput and cost, but users are forgiving if they see progress
Streaming responses (displaying tokens as they arrive) exploits this psychology—users read while generation continues, making even 30-second responses feel acceptable.
Optimization Implications
Improving TTFT requires different techniques than reducing total time:
- TTFT: prompt caching, faster first-pass inference, prefill optimization
- Total time: quantization, batch processing, better hardware
Key Takeaway: TTFT governs perceived responsiveness and user satisfaction, while total generation time affects system throughput—optimize TTFT first for better UX, even if it doesn't reduce actual compute time.