How Large Language Models Work
Tokens, transformers, attention, context windows, training stages (pre-training, fine-tuning, RLHF), inference parameters, and architect-relevant limits like latency, cost and hallucination.
Key concepts
- Tokens: text is split into sub-word tokens (roughly ¾ of an English word on average). Pricing, limits and latency are per token.
- Transformer: neural architecture using self-attention so each token can attend to every other token in the context (Vaswani et al., 2017).
- Context window: maximum tokens per request (input + output). Modern models support hundreds of thousands to over a million tokens (version-sensitive), but quality and cost degrade with very long contexts.
- Decoder-only generation: the model predicts the next token repeatedly (autoregressive), so output length drives latency.
- KV cache: stores attention keys/values for processed tokens to speed generation; prompt caching reuses it across requests with identical prefixes.
Training stages
| Stage | What happens | Who does it |
|---|---|---|
| Pre-training | Next-token prediction on trillions of tokens | Model providers |
| Supervised fine-tuning (SFT) | Learn to follow instructions from examples | Providers; enterprises for specialization |
| Preference alignment (RLHF, RLAIF, DPO) | Optimize for helpful, harmless responses | Providers |
| Domain fine-tuning / LoRA adapters | Adapt style, format or narrow tasks | Enterprises (selectively) |
flowchart LR PT[Pre-training<br/>web, books, code] --> BASE[Base model] BASE --> SFT[Instruction tuning] SFT --> ALIGN[Preference alignment<br/>RLHF / DPO] ALIGN --> CHAT[Chat / instruct model] CHAT --> FT[Optional enterprise fine-tune<br/>LoRA, distillation] CHAT --> RAG[RAG + tools at inference]
Inference parameters
| Parameter | Effect |
|---|---|
| Temperature | Randomness; low (0–0.3) for factual/extraction tasks |
| Top-p / top-k | Sampling restriction |
| Max output tokens | Caps length, cost and latency |
| Stop sequences | End generation |
| Structured output / JSON schema | Constrains output format |
Limits architects must design around
| Limit | Design response |
|---|---|
| Hallucination | Grounding with RAG, citations, verification, evaluation |
| Knowledge cutoff | Retrieval and tools for fresh data |
| Latency (time to first token, tokens/s) | Streaming, smaller models, caching, parallel calls |
| Cost per token | Model routing, prompt caching, shorter contexts |
| Non-determinism | Low temperature, structured outputs, evaluation suites |
| Prompt injection | Guardrails, least-privilege tools |
Fine-tune or not?
Use prompting + RAG first. Fine-tune when you need consistent style/format, a narrow task at lower cost/latency (smaller tuned model), or domain language the base model handles poorly. Fine-tuning does not reliably add facts and does not respect document permissions.
Interview questions and answers
Why do LLMs hallucinate?
They generate statistically likely text, not verified facts. Without relevant context, they fill gaps plausibly. Grounding, refusal instructions and verification reduce but do not eliminate it.
How do you estimate the cost of an LLM feature?
Requests per day × (input tokens × input price + output tokens × output price), adjusted for caching and routing; add embedding, vector store and guardrail costs; compare to the value per request.