Course contentsShow
← AI Engineering
Lesson 1011 of 1,88625. Model Serving and Inference OptimizationFree lesson

vLLM Deployment Patterns

Deploying vLLM: OpenAI-compatible API, configuration options, and model loading strategies.

vLLM Deployment Patterns

What You'll Learn

You'll learn how to deploy vLLM with an OpenAI-compatible API, configure key serving options, and choose the right model loading strategy for your use case.

The Core Idea

vLLM isn't just a serving framework—it's designed to be production-ready out of the box. One of its killer features is OpenAI API compatibility, meaning you can swap out OpenAI calls with your self-hosted vLLM endpoint with minimal code changes. Think of it as running your own ChatGPT-style API server, complete with streaming support and familiar request/response formats.

OpenAI-Compatible API

Start vLLM's API server with:

python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Llama-2-7b-hf \
  --host 0.0.0.0 \
  --port 8000

This creates endpoints like /v1/completions and /v1/chat/completions that work with OpenAI client libraries.

Configuration Options

Key flags control performance and resource usage:

  • --tensor-parallel-size: Split model across GPUs (for large models)
  • --max-num-seqs: Maximum concurrent sequences (controls throughput vs latency)
  • --gpu-memory-utilization: Fraction of GPU memory to use (default 0.9)
  • --dtype: Precision (float16, bfloat16, or auto)

Model Loading Strategies

Eager loading (default): Load the entire model at startup—slower start, faster inference.

Lazy loading: Load model weights on first request—faster deployment, one-time latency hit.

Quantized models: Load INT8/INT4 versions for memory efficiency using --quantization awq or similar flags.

Key Takeaway: vLLM's OpenAI-compatible API server combines familiar developer experience with flexible configuration options—choose tensor parallelism for large models, tune concurrency limits for your workload, and select loading strategies that balance startup time against serving performance.