Course contentsShow
← AI Engineering
Lesson 26 of 1,8861. AI Engineering Mindset and LandscapeFree lesson

Latency and Performance Requirements

Analyzing how response time needs affect choices between managed APIs, edge deployment, and local inference.

Latency and Performance Requirements

What you'll learn: How the speed your AI system needs to respond directly shapes whether you use cloud APIs, edge devices, or local inference.

The Speed Constraint

In AI Engineering, latency means how long it takes from sending a request to getting an answer back. Different applications have wildly different needs:

  • A chatbot might tolerate 2-3 seconds
  • A content recommendation engine can wait 500ms
  • An autonomous vehicle needs answers in under 100ms
  • Real-time video processing might demand under 50ms

These requirements force your architecture choices.

The Three Deployment Options

Managed APIs (like OpenAI's GPT-4 API) are convenient but add network round-trip time—typically 200-1000ms just for data travel, plus processing time. Fine for chatbots, problematic for real-time systems.

Edge deployment puts models on devices closer to users—think smartphones or IoT devices. This cuts network latency dramatically but limits you to smaller models that fit on constrained hardware.

Local inference runs models on dedicated servers you control. You get predictable performance and can optimize hardware for your exact workload, but you handle all the infrastructure complexity yourself.

Think Like a Restaurant

Imagine ordering food:

  • API: Delivery service (convenient, but takes 30-45 minutes)
  • Edge: Microwave meal at home (fast, but limited menu)
  • Local: Your own kitchen (full control, fastest for repeated meals, but you buy equipment and ingredients)

Your performance requirements are like how hungry you are and when you need to eat.

Key Takeaway: The milliseconds you have to respond determine your deployment strategy—APIs offer simplicity but add latency, edge cuts response time but limits model size, and local inference gives control at the cost of operational complexity.