Production GenAI Reference Architecture

An enterprise reference architecture for GenAI applications covering experience, orchestration, retrieval, models, tools, guardrails, evaluation, observability and governance layers.

Layers

flowchart TB
  subgraph Experience
    UI[Chat UI, copilots in apps, APIs, Teams/Slack bots]
  end
  subgraph Orchestration
    ORCH[Application orchestrator<br/>workflows, agents, prompts]
    MEMO[(Conversation & memory)]
  end
  subgraph Knowledge
    ING[Ingestion & connectors] --> IDX[(Hybrid index + ACLs)]
    KG[(Knowledge graph, optional)]
  end
  subgraph Models
    GW[LLM gateway: routing, quotas, caching]
    FM[Foundation models: managed APIs]
    OSS[Self-hosted / fine-tuned models]
    EMB[Embedding & rerank models]
  end
  subgraph Tools
    MCP[MCP servers / tool gateway]
    SYS[Enterprise systems]
  end
  subgraph Trust
    GR[Guardrails]
    EVAL[Evaluation pipelines]
    OBS[Observability & cost]
    GOV[Governance: registry, policies, audit]
  end
  UI --> ORCH
  ORCH --> MEMO
  ORCH --> IDX
  ORCH --> GW
  GW --> FM & OSS
  ING --> EMB
  ORCH --> MCP --> SYS
  ORCH --> GR
  ORCH --> OBS
  EVAL --> ORCH
  GOV -.-> ORCH

Platform vs application responsibilities

Platform team provides Application teams provide
LLM gateway, approved models, credentials Use-case design, prompts, workflows
RAG building blocks, connectors, vector stores Domain content curation, chunking choices
Guardrail services and policies Use-case-specific guardrail configuration
Evaluation tooling, observability, cost dashboards Golden datasets, success metrics
MCP gateway and shared tool servers Domain-specific tools with proper authorization
Governance workflow and registry Risk assessment inputs, documentation

Non-functional considerations

  • Latency: streaming, parallel retrieval, small models for intermediate steps.
  • Availability: multi-model/multi-region fallback via gateway.
  • Cost: budgets per team, routing, caching.
  • Security: identity propagation to tools and retrieval, data residency, no training on enterprise data.
  • Portability: abstract model providers behind the gateway; keep prompts/evals provider-neutral where practical.

Interview questions and answers

What would you build first for a company starting with GenAI?

An LLM gateway with approved models and logging, a secure chat assistant over a few curated sources with RAG and citations, an evaluation and observability baseline, and a lightweight governance process. Then scale RAG and add tools based on proven use cases.