Async Batch Processing with asyncio
You'll learn how to use Python's asyncio to send multiple embedding or API requests concurrently, dramatically speeding up data pipeline processing.
The Problem with Sequential Requests
When you send API requests one at a time (synchronously), each request must complete before the next begins. If each embedding API call takes 200ms and you have 1,000 documents, you'll wait 200 seconds—over 3 minutes—just sitting idle while network I/O completes.
The Asynchronous Solution
Asyncio allows you to fire off many requests simultaneously without waiting for each to finish. Think of it like a restaurant: instead of one waiter taking an order, waiting in the kitchen until it's ready, then taking the next order (synchronous), you have waiters who take multiple orders and check back when food is ready (asynchronous).
When a request is waiting for the API server to respond, your program can send more requests. This concurrent approach maximizes throughput by keeping network connections busy.
How It Works
You define functions with async def and use await when making I/O calls (like HTTP requests). The asyncio.gather() or asyncio.create_task() functions let you run multiple async operations concurrently:
import asyncio
async def embed_batch(texts):
# Create tasks for all API calls at once
tasks = [call_embedding_api_async(text) for text in texts]
# Run them concurrently and wait for all to complete
return await asyncio.gather(*tasks)
Instead of 200 seconds, you might process those 1,000 documents in 5-10 seconds, limited mainly by API rate limits rather than sequential waiting.
Key Takeaway: Asyncio enables concurrent API requests that run "at the same time," reducing total pipeline execution time from hours to minutes by eliminating idle waiting between I/O operations.