Course contentsShow
← AI Engineering
Lesson 537 of 1,88613. LLM Orchestration FrameworksFree lesson

Performance Comparison: Framework vs Raw

Measuring latency, overhead, and resource usage differences between framework-wrapped and direct API calls.

Performance Comparison: Framework vs Raw

What You'll Learn

You'll learn how to properly measure and understand the real-world performance differences between using an abstraction framework like LangChain and making direct API calls.

Measuring the Overhead

When you wrap API calls in a framework, you gain convenience but introduce abstraction layers. To understand the actual execution cost, you must measure the latency properly.

A naive benchmark might simply run one request per path. However, cold starts, variable network conditions, and internal retry logic can heavily skew single-sample measurements. For a fair comparison, you must:

  • Warm up both paths: Run an initial throwaway call to establish connections and load background libraries.
  • Fix output length and parameters: Set a low, fixed max_tokens and temperature=0 so generation time is deterministic and consistent.
  • Disable retries: Ensure a temporary network hiccup isn't masked as framework latency by setting max_retries=0.
  • Run repeated samples: Collect multiple timings using time.perf_counter and compare both the median and the spread (standard deviation).

Practical Measurement Approach

Here is a rigorous way to benchmark direct API calls versus a framework in Python:

import os
import statistics
import time
from openai import OpenAI
from langchain_openai import ChatOpenAI
from langchain_core.messages import HumanMessage

# Take the model name from an environment variable
MODEL_NAME = os.environ["OPENAI_MODEL"]

# 1. Initialize clients with 0 retries for fairness
client = OpenAI(max_retries=0)
lc_llm = ChatOpenAI(
    model=MODEL_NAME,
    max_retries=0,
    temperature=0,
    max_tokens=5
)

raw_msgs = [{"role": "user", "content": "Say hi."}]
lc_msgs = [HumanMessage(content="Say hi.")]

def time_raw():
    start = time.perf_counter()
    client.chat.completions.create(
        model=MODEL_NAME,
        messages=raw_msgs,
        max_tokens=5,
        temperature=0
    )
    return time.perf_counter() - start

def time_langchain():
    start = time.perf_counter()
    lc_llm.invoke(lc_msgs)
    return time.perf_counter() - start

# 2. Warm up both paths
time_raw()
time_langchain()

# 3. Collect repeated samples
SAMPLES = 20
raw_times = [time_raw() for _ in range(SAMPLES)]
lc_times = [time_langchain() for _ in range(SAMPLES)]

print(f"Raw API median: {statistics.median(raw_times):.4f}s "
      f"(spread: {statistics.stdev(raw_times):.4f}s)")
print(f"LangChain median: {statistics.median(lc_times):.4f}s "
      f"(spread: {statistics.stdev(lc_times):.4f}s)")

If you run this benchmark, you will observe that network and model generation latency usually dominate the total response time. Frameworks do add baseline CPU and memory overhead, but you must measure your own setup in conditions mimicking production to decide if the developer productivity benefits justify the extra layers of execution.

Key takeaway: Proper benchmarking requires warming up connections, repeated sampling, and fixing generation parameters; while frameworks add abstraction layers, network and model latency usually dominate total response times.