Async/Await and Concurrent API Calls
What you'll learn: How to use Python's asyncio to make multiple LLM API calls at the same time, dramatically reducing total wait time.
The Waiting Problem
When you make LLM API calls sequentially (one after another), you spend most of your time waiting for responses. If each call takes 2 seconds and you need 5 calls, you're waiting 10 seconds total—even though your CPU is idle most of that time.
The solution? Make those calls concurrently using async/await.
How Async/Await Works
Think of it like ordering coffee for a group. Sequential means you wait for each person's drink before ordering the next one. Concurrent means you place all orders at once and collect them as they're ready—much faster.
In Python, you mark functions that can wait with async def, and you use await when calling them:
import asyncio
import httpx # async-friendly HTTP client
async def call_llm(prompt):
async with httpx.AsyncClient() as client:
response = await client.post(
"https://api.provider.com/v1/completions",
json={"prompt": prompt}
)
return response.json()
async def main():
# Run 3 calls concurrently
results = await asyncio.gather(
call_llm("Summarize topic A"),
call_llm("Summarize topic B"),
call_llm("Summarize topic C")
)
return results
# Execute
asyncio.run(main())
asyncio.gather() launches all calls simultaneously. Instead of 6 seconds (3 × 2s), you wait ~2 seconds total.
When to Use This
Perfect for parallelizable operations you identified earlier: multiple document summaries, batch classifications, or independent retrieval queries. Don't use it when calls depend on each other's results.
Key Takeaway: Async/await with asyncio.gather() lets you execute multiple independent LLM calls in parallel, reducing latency from the sum of all calls to roughly the duration of the slowest single call.