Course contentsShow
← AI Engineering
Lesson 480 of 1,88612. Data Pipelines for AI SystemsFree lesson

Batching Requests to Embedding APIs

Optimizing throughput by batching multiple texts into single API calls while respecting token and array limits.

Batching Requests to Embedding APIs

What you'll learn: How to maximize efficiency by sending multiple texts in a single API call instead of making hundreds of individual requests.

The Core Problem

When embedding thousands of documents, making one API call per text is painfully slow and wastes resources. Each HTTP request has overhead—establishing connections, waiting for responses, and dealing with network latency. It's like driving to the grocery store separately for each item instead of buying everything in one trip.

How Batching Works

Most embedding APIs (OpenAI, Cohere, etc.) accept arrays of texts in a single request. Instead of:

# Inefficient: 3 separate API calls
embedding1 = embed_api.create(input="Text 1")
embedding2 = embed_api.create(input="Text 2")
embedding3 = embed_api.create(input="Text 3")

You send:

# Efficient: 1 API call for 3 texts
embeddings = embed_api.create(input=["Text 1", "Text 2", "Text 3"])

Constraints to Respect

APIs impose limits you must honor:

  • Array size limits: Maximum number of texts per batch (e.g., 2048 items)
  • Total token limits: Combined token count across all texts (e.g., 8,191 tokens)
  • Rate limits: Requests per minute (already covered in lesson 479)

Your batching logic should group texts until you approach—but don't exceed—these boundaries, then send the batch and start a new one.

The Efficiency Gain

Batching 100 texts into 5 calls of 20 each is 20× faster than 100 individual calls, thanks to reduced overhead. You're maximizing the "payload" of each API interaction.

Key Takeaway: Batching multiple texts into single embedding API calls dramatically improves throughput by reducing network overhead, as long as you respect per-request array and token limits.