Model Serving and AI Orchestration
Serving options from managed APIs to self-hosted GPUs, inference optimization (batching, quantization, KV cache), LLM gateways, routing, caching and capacity planning.
Serving options
| Option | Pros | Cons |
|---|---|---|
| Managed model APIs (Bedrock, Azure OpenAI/Foundry, Vertex AI, provider APIs) | No infrastructure, latest models, enterprise terms | Per-token cost, rate limits, less control |
| Provisioned throughput on managed services | Guaranteed capacity, predictable latency | Commitment cost |
| Self-hosted open models (vLLM, TGI, SGLang, TensorRT-LLM, Triton on GPUs) | Control, data locality, cost at high utilization | GPU scarcity, MLOps skills, model quality gap vs frontier models |
| Classical ML serving (SageMaker, Vertex endpoints, KServe, BentoML, in-process ONNX) | Cheap, low latency | Separate lifecycle |
Inference optimization
- Continuous batching (vLLM, TGI): combine requests dynamically for GPU efficiency.
- PagedAttention / KV cache management: more concurrent sequences per GPU.
- Quantization (8-bit, 4-bit, FP8): smaller memory, faster, some quality loss.
- Speculative decoding: a small draft model proposes tokens verified by the large model.
- Prompt/prefix caching: reuse computation for shared prefixes.
- Distillation: train a smaller model to mimic a larger one for a narrow task.
LLM gateway
flowchart LR APPS[Applications] --> GW[LLM gateway] GW --> AUTH[AuthN, per-team quotas, budgets] GW --> ROUTE[Routing: task → model<br/>fallback across providers/regions] GW --> CACHE[Exact + semantic cache] GW --> GR[Guardrails] GW --> OBS[Logging, tracing, cost attribution] ROUTE --> M1[Frontier model API] ROUTE --> M2[Small fast model] ROUTE --> M3[Self-hosted open model]
A gateway (LiteLLM, Portkey, Kong AI Gateway, cloud-native options such as Azure API Management AI gateway policies) centralizes credentials, quotas, routing, fallbacks, caching and cost tracking, and makes model changes transparent to applications.
Capacity planning for self-hosting
Estimate tokens/s needed at peak (requests/s × output tokens), measure throughput per GPU for your model and batch sizes, include prompt processing cost (prefill), and keep headroom for latency targets. GPU memory must hold model weights + KV cache for concurrent sequences.
Interview questions and answers
When does self-hosting an open model make sense?
High, steady volume where GPU utilization stays high; strict data residency or air-gapped requirements; need for customization (fine-tuning, specialized models); or latency requirements a managed API cannot meet. Otherwise managed APIs are usually cheaper in total cost and faster to evolve.