Course contentsShow
← AI Engineering
Lesson 38 of 1,8861. AI Engineering Mindset and LandscapeFree lesson

Building Cost into Architecture Decisions

Make architectural choices with cost awareness: when to cache, batch, or use embeddings versus LLM calls.

Building Cost into Architecture Decisions

What you'll learn: How to make architectural choices that balance functionality and cost from day one, not as an afterthought.

The Core Idea

When designing AI systems, every architectural decision has a cost implication. The key is asking "what will this cost?" before you commit to a pattern, not after your bill arrives. This means actively choosing between strategies like caching responses, batching requests, or using embeddings instead of expensive LLM calls.

Think of it like planning a road trip: you could drive straight through (fast but expensive in gas and tolls), or you could plan stops, carpool legs, and use fuel-efficient routes. Both get you there, but the cost difference is massive.

Key Architectural Cost Questions

Should I cache this? If users ask similar questions repeatedly, storing responses can eliminate 70%+ of LLM calls. The cost of a cache lookup is pennies compared to a GPT-4 call.

Can I batch requests? Processing 10 requests at once instead of individually often reduces costs through efficiency gains, especially for embedding generation or fine-tuning jobs.

Do I need an LLM call here? Sometimes a simple embedding similarity search (finding related documents) costs 100x less than asking an LLM to do the same task. If you're matching, comparing, or searching, embeddings might be your answer.

What's my fallback strategy? Maybe you use a cheaper model for most requests and only call the expensive one when confidence is low.

The goal isn't always "cheapest possible"—it's making informed choices where you understand the cost-performance trade-off before building.

Key Takeaway: Design your architecture with cost as a first-class constraint alongside performance and accuracy—choose caching, batching, and embeddings strategically to avoid expensive surprises.