Polling Patterns and Best Practices
What you'll learn: How to implement smart polling so clients can efficiently check if long-running LLM jobs are complete without overwhelming your servers.
What Is Polling?
When a client submits a long-running LLM request (like generating a 5,000-word document or processing hundreds of embeddings), it receives a job ID and must periodically ask the server: "Is it done yet?" That's polling. Done naively, you create two problems: clients hammer your server every second, and users stare at frozen loading spinners.
Exponential Backoff: The Core Pattern
Instead of checking every second forever, exponential backoff increases wait time between checks:
Check at: 1s → 2s → 4s → 8s → 16s → 30s (capped)
This works like checking cookies baking in an oven. You peek after 1 minute, then 2, then 5—you don't open the oven every 10 seconds.
Status Endpoint Design
Create a dedicated status endpoint that returns the job state:
GET /jobs/{job_id}/status
→ { "status": "processing", "progress": 45, "estimated_seconds": 120 }
Return actionable states: queued, processing, completed, failed. Include progress indicators and time estimates when possible.
Best Practices Summary
- Cap your backoff at a reasonable maximum (e.g., 30-60 seconds)
- Add jitter (random variance) to prevent thundering herd when many jobs complete simultaneously
- Include retry-after headers to tell clients when to check next
- Time out gracefully after a maximum number of attempts
- Provide webhooks as an alternative for clients who support them
Key Takeaway: Exponential backoff with status endpoints balances server load with user experience—clients check frequently at first when jobs complete quickly, then back off for longer operations.