Course contentsShow
← AI Engineering
Lesson 1136 of 1,88628. Prompt and Chain OptimizationFree lesson

Time-to-First-Token vs Total Generation Time

Distinguish TTFT from complete response latency and understand their different user experience impacts.

Time-to-First-Token vs Total Generation Time

What you'll learn: How to distinguish between the user's perceived responsiveness and the actual completion time of LLM responses.

When an LLM generates text, two critical latency metrics matter for user experience:

Time-to-First-Token (TTFT)

This measures how long from your request until the first token appears. It includes:

  • Prompt processing time
  • Model inference startup
  • Network overhead

TTFT determines how quickly users see something happening. Even if the full response takes 10 seconds, seeing the first word in 200ms makes the system feel responsive.

Total Generation Time

This measures from request to complete response. It includes:

  • TTFT
  • Plus all subsequent token generation
  • Streaming duration

This is your actual end-to-end latency, but matters less for perceived speed.

Why They Impact UX Differently

Think of it like streaming video: TTFT is like how fast the video starts playing (buffering time), while total generation time is the entire video duration. Users tolerate long videos if they start instantly, but hate short videos that buffer forever.

For chatbots and assistants:

  • TTFT < 300ms feels instant and responsive
  • TTFT > 2 seconds feels broken, even if total time is reasonable
  • Total time matters for throughput and cost, but users are forgiving if they see progress

Streaming responses (displaying tokens as they arrive) exploits this psychology—users read while generation continues, making even 30-second responses feel acceptable.

Optimization Implications

Improving TTFT requires different techniques than reducing total time:

  • TTFT: prompt caching, faster first-pass inference, prefill optimization
  • Total time: quantization, batch processing, better hardware

Key Takeaway: TTFT governs perceived responsiveness and user satisfaction, while total generation time affects system throughput—optimize TTFT first for better UX, even if it doesn't reduce actual compute time.