An LLM feature costs $0.30 per request and takes 12 seconds. How do you bring it to $0.03 and 3 seconds?

Short answer

Measure where tokens and time go, then reduce context (better retrieval and reranking, summarizing history), route simple requests to smaller models, use prompt caching for stable prefixes, stream output, parallelize independent steps, cap output length, cache frequent answers, and consider fine-tuned or distilled small models for narrow tasks, validating quality with evals at each step.

Detailed answer

Lever Cost impact Latency impact
Trim context (fewer, better chunks; summarized history) High High
Model routing (small model for easy/intermediate steps) High High
Prompt caching Medium–high on long prefixes Medium
Streaming None Perceived latency much lower
Parallel tool/retrieval calls None High
Max output tokens / concise formats Medium Medium
Semantic/exact answer cache High for repeated questions High
Fine-tuned small model / distillation High at volume High
Batch API for offline workloads High n/a

Validate every change against the evaluation set; track cost per successful outcome.

What the interviewer expects

A measurement-driven, multi-lever plan that protects quality.

Common mistakes to avoid

Cutting context blindly and degrading accuracy; ignoring time-to-first-token vs total latency.