How Large Language Models Work

Tokens, transformers, attention, context windows, training stages (pre-training, fine-tuning, RLHF), inference parameters, and architect-relevant limits like latency, cost and hallucination.

Key concepts

  • Tokens: text is split into sub-word tokens (roughly ¾ of an English word on average). Pricing, limits and latency are per token.
  • Transformer: neural architecture using self-attention so each token can attend to every other token in the context (Vaswani et al., 2017).
  • Context window: maximum tokens per request (input + output). Modern models support hundreds of thousands to over a million tokens (version-sensitive), but quality and cost degrade with very long contexts.
  • Decoder-only generation: the model predicts the next token repeatedly (autoregressive), so output length drives latency.
  • KV cache: stores attention keys/values for processed tokens to speed generation; prompt caching reuses it across requests with identical prefixes.

Training stages

Stage What happens Who does it
Pre-training Next-token prediction on trillions of tokens Model providers
Supervised fine-tuning (SFT) Learn to follow instructions from examples Providers; enterprises for specialization
Preference alignment (RLHF, RLAIF, DPO) Optimize for helpful, harmless responses Providers
Domain fine-tuning / LoRA adapters Adapt style, format or narrow tasks Enterprises (selectively)
flowchart LR
  PT[Pre-training<br/>web, books, code] --> BASE[Base model]
  BASE --> SFT[Instruction tuning]
  SFT --> ALIGN[Preference alignment<br/>RLHF / DPO]
  ALIGN --> CHAT[Chat / instruct model]
  CHAT --> FT[Optional enterprise fine-tune<br/>LoRA, distillation]
  CHAT --> RAG[RAG + tools at inference]

Inference parameters

Parameter Effect
Temperature Randomness; low (0–0.3) for factual/extraction tasks
Top-p / top-k Sampling restriction
Max output tokens Caps length, cost and latency
Stop sequences End generation
Structured output / JSON schema Constrains output format

Limits architects must design around

Limit Design response
Hallucination Grounding with RAG, citations, verification, evaluation
Knowledge cutoff Retrieval and tools for fresh data
Latency (time to first token, tokens/s) Streaming, smaller models, caching, parallel calls
Cost per token Model routing, prompt caching, shorter contexts
Non-determinism Low temperature, structured outputs, evaluation suites
Prompt injection Guardrails, least-privilege tools

Fine-tune or not?

Use prompting + RAG first. Fine-tune when you need consistent style/format, a narrow task at lower cost/latency (smaller tuned model), or domain language the base model handles poorly. Fine-tuning does not reliably add facts and does not respect document permissions.

Interview questions and answers

Why do LLMs hallucinate?

They generate statistically likely text, not verified facts. Without relevant context, they fill gaps plausibly. Grounding, refusal instructions and verification reduce but do not eliminate it.

How do you estimate the cost of an LLM feature?

Requests per day × (input tokens × input price + output tokens × output price), adjusted for caching and routing; add embedding, vector store and guardrail costs; compare to the value per request.