Model Serving and AI Orchestration

Serving options from managed APIs to self-hosted GPUs, inference optimization (batching, quantization, KV cache), LLM gateways, routing, caching and capacity planning.

Serving options

Option Pros Cons
Managed model APIs (Bedrock, Azure OpenAI/Foundry, Vertex AI, provider APIs) No infrastructure, latest models, enterprise terms Per-token cost, rate limits, less control
Provisioned throughput on managed services Guaranteed capacity, predictable latency Commitment cost
Self-hosted open models (vLLM, TGI, SGLang, TensorRT-LLM, Triton on GPUs) Control, data locality, cost at high utilization GPU scarcity, MLOps skills, model quality gap vs frontier models
Classical ML serving (SageMaker, Vertex endpoints, KServe, BentoML, in-process ONNX) Cheap, low latency Separate lifecycle

Inference optimization

  • Continuous batching (vLLM, TGI): combine requests dynamically for GPU efficiency.
  • PagedAttention / KV cache management: more concurrent sequences per GPU.
  • Quantization (8-bit, 4-bit, FP8): smaller memory, faster, some quality loss.
  • Speculative decoding: a small draft model proposes tokens verified by the large model.
  • Prompt/prefix caching: reuse computation for shared prefixes.
  • Distillation: train a smaller model to mimic a larger one for a narrow task.

LLM gateway

flowchart LR
  APPS[Applications] --> GW[LLM gateway]
  GW --> AUTH[AuthN, per-team quotas, budgets]
  GW --> ROUTE[Routing: task → model<br/>fallback across providers/regions]
  GW --> CACHE[Exact + semantic cache]
  GW --> GR[Guardrails]
  GW --> OBS[Logging, tracing, cost attribution]
  ROUTE --> M1[Frontier model API]
  ROUTE --> M2[Small fast model]
  ROUTE --> M3[Self-hosted open model]

A gateway (LiteLLM, Portkey, Kong AI Gateway, cloud-native options such as Azure API Management AI gateway policies) centralizes credentials, quotas, routing, fallbacks, caching and cost tracking, and makes model changes transparent to applications.

Capacity planning for self-hosting

Estimate tokens/s needed at peak (requests/s × output tokens), measure throughput per GPU for your model and batch sizes, include prompt processing cost (prefill), and keep headroom for latency targets. GPU memory must hold model weights + KV cache for concurrent sequences.

Interview questions and answers

When does self-hosting an open model make sense?

High, steady volume where GPU utilization stays high; strict data residency or air-gapped requirements; need for customization (fine-tuning, specialized models); or latency requirements a managed API cannot meet. Otherwise managed APIs are usually cheaper in total cost and faster to evolve.