An LLM feature costs $0.30 per request and takes 12 seconds. How do you bring it to $0.03 and 3 seconds?
Short answer
Measure where tokens and time go, then reduce context (better retrieval and reranking, summarizing history), route simple requests to smaller models, use prompt caching for stable prefixes, stream output, parallelize independent steps, cap output length, cache frequent answers, and consider fine-tuned or distilled small models for narrow tasks, validating quality with evals at each step.
Detailed answer
| Lever | Cost impact | Latency impact |
|---|---|---|
| Trim context (fewer, better chunks; summarized history) | High | High |
| Model routing (small model for easy/intermediate steps) | High | High |
| Prompt caching | Medium–high on long prefixes | Medium |
| Streaming | None | Perceived latency much lower |
| Parallel tool/retrieval calls | None | High |
| Max output tokens / concise formats | Medium | Medium |
| Semantic/exact answer cache | High for repeated questions | High |
| Fine-tuned small model / distillation | High at volume | High |
| Batch API for offline workloads | High | n/a |
Validate every change against the evaluation set; track cost per successful outcome.
What the interviewer expects
A measurement-driven, multi-lever plan that protects quality.
Common mistakes to avoid
Cutting context blindly and degrading accuracy; ignoring time-to-first-token vs total latency.