Course contentsShow
← AI Engineering
Lesson 937 of 1,88623. LLM Application Architecture PatternsFree lesson

Polling Patterns and Best Practices

Implementing exponential backoff and status endpoints for clients to check long-running job completion.

Polling Patterns and Best Practices

What you'll learn: How to implement smart polling so clients can efficiently check if long-running LLM jobs are complete without overwhelming your servers.

What Is Polling?

When a client submits a long-running LLM request (like generating a 5,000-word document or processing hundreds of embeddings), it receives a job ID and must periodically ask the server: "Is it done yet?" That's polling. Done naively, you create two problems: clients hammer your server every second, and users stare at frozen loading spinners.

Exponential Backoff: The Core Pattern

Instead of checking every second forever, exponential backoff increases wait time between checks:

Check at: 1s → 2s → 4s → 8s → 16s → 30s (capped)

This works like checking cookies baking in an oven. You peek after 1 minute, then 2, then 5—you don't open the oven every 10 seconds.

Status Endpoint Design

Create a dedicated status endpoint that returns the job state:

GET /jobs/{job_id}/status
→ { "status": "processing", "progress": 45, "estimated_seconds": 120 }

Return actionable states: queued, processing, completed, failed. Include progress indicators and time estimates when possible.

Best Practices Summary

  • Cap your backoff at a reasonable maximum (e.g., 30-60 seconds)
  • Add jitter (random variance) to prevent thundering herd when many jobs complete simultaneously
  • Include retry-after headers to tell clients when to check next
  • Time out gracefully after a maximum number of attempts
  • Provide webhooks as an alternative for clients who support them

Key Takeaway: Exponential backoff with status endpoints balances server load with user experience—clients check frequently at first when jobs complete quickly, then back off for longer operations.