Production GenAI Reference Architecture
An enterprise reference architecture for GenAI applications covering experience, orchestration, retrieval, models, tools, guardrails, evaluation, observability and governance layers.
Layers
flowchart TB
subgraph Experience
UI[Chat UI, copilots in apps, APIs, Teams/Slack bots]
end
subgraph Orchestration
ORCH[Application orchestrator<br/>workflows, agents, prompts]
MEMO[(Conversation & memory)]
end
subgraph Knowledge
ING[Ingestion & connectors] --> IDX[(Hybrid index + ACLs)]
KG[(Knowledge graph, optional)]
end
subgraph Models
GW[LLM gateway: routing, quotas, caching]
FM[Foundation models: managed APIs]
OSS[Self-hosted / fine-tuned models]
EMB[Embedding & rerank models]
end
subgraph Tools
MCP[MCP servers / tool gateway]
SYS[Enterprise systems]
end
subgraph Trust
GR[Guardrails]
EVAL[Evaluation pipelines]
OBS[Observability & cost]
GOV[Governance: registry, policies, audit]
end
UI --> ORCH
ORCH --> MEMO
ORCH --> IDX
ORCH --> GW
GW --> FM & OSS
ING --> EMB
ORCH --> MCP --> SYS
ORCH --> GR
ORCH --> OBS
EVAL --> ORCH
GOV -.-> ORCHPlatform vs application responsibilities
| Platform team provides | Application teams provide |
|---|---|
| LLM gateway, approved models, credentials | Use-case design, prompts, workflows |
| RAG building blocks, connectors, vector stores | Domain content curation, chunking choices |
| Guardrail services and policies | Use-case-specific guardrail configuration |
| Evaluation tooling, observability, cost dashboards | Golden datasets, success metrics |
| MCP gateway and shared tool servers | Domain-specific tools with proper authorization |
| Governance workflow and registry | Risk assessment inputs, documentation |
Non-functional considerations
- Latency: streaming, parallel retrieval, small models for intermediate steps.
- Availability: multi-model/multi-region fallback via gateway.
- Cost: budgets per team, routing, caching.
- Security: identity propagation to tools and retrieval, data residency, no training on enterprise data.
- Portability: abstract model providers behind the gateway; keep prompts/evals provider-neutral where practical.
Interview questions and answers
What would you build first for a company starting with GenAI?
An LLM gateway with approved models and logging, a secure chat assistant over a few curated sources with RAG and citations, an evaluation and observability baseline, and a lightweight governance process. Then scale RAG and add tools based on proven use cases.