Spring AI for Java Architects

Building LLM-powered features in Java with Spring AI's ChatClient, prompt templates, structured output, tool calling, RAG with vector stores, MCP and observability.

[!VERSION] Spring AI reached 1.0 GA in May 2025, followed by 1.x releases (with MCP client/server support). APIs evolve quickly; verify against the current reference documentation.

What Spring AI provides

  • A portable ChatClient API across providers (OpenAI, Azure OpenAI, Anthropic, Amazon Bedrock, Google Vertex AI/Gemini, Ollama, Mistral and others).
  • Prompt templates, structured output conversion into Java records.
  • Tool (function) calling with Java methods.
  • Embedding models and a VectorStore abstraction (pgvector, Redis, OpenSearch, Elasticsearch, MongoDB Atlas, Pinecone, Qdrant, Weaviate and more).
  • Advisors for RAG, chat memory and safety.
  • MCP client and server starters.
  • Observability via Micrometer (token usage, latency).

Chat with structured output

record TicketTriage(String category, String priority, String summary) {}

@Service
class TriageService {
    private final ChatClient chat;
    TriageService(ChatClient.Builder builder) {
        this.chat = builder.defaultSystem("You classify IT support tickets. Use only the given categories.").build();
    }

    TicketTriage triage(String ticketText) {
        return chat.prompt()
            .user(u -> u.text("Categories: ACCESS, HARDWARE, NETWORK, SOFTWARE. Ticket: {t}").param("t", ticketText))
            .call()
            .entity(TicketTriage.class);    // JSON schema derived from the record
    }
}

Tool calling

class OrderTools {
    private final OrderService orders;
    OrderTools(OrderService orders) { this.orders = orders; }

    @Tool(description = "Get the status of an order owned by the current user")
    OrderStatus orderStatus(@ToolParam(description = "Order ID") String orderId) {
        return orders.statusForCurrentUser(orderId);   // authorization enforced here, not by the model
    }
}

String answer = chat.prompt().user("Where is my order 8812?").tools(new OrderTools(orderService)).call().content();

RAG with a vector store

@Bean
ChatClient ragClient(ChatClient.Builder builder, VectorStore store) {
    return builder
        .defaultAdvisors(QuestionAnswerAdvisor.builder(store)
            .searchRequest(SearchRequest.builder().topK(6).similarityThreshold(0.6).build())
            .build())
        .build();
}

Add metadata filters for permissions (e.g. filterExpression("department in ['hr','all']")) derived from the authenticated user.

Architect-level considerations

[!ARCHITECT] Spring AI is a good fit when the enterprise standard is JVM and AI features live inside existing Spring services. For heavy experimentation, evaluation pipelines and the newest research techniques, Python ecosystems move faster; many organizations combine Java services with Python AI services behind APIs.

[!WARNING] Never trust model output for authorization or as executable input. Tools must enforce permissions and validate parameters.

Interview questions and answers

How would you add GenAI to an existing Spring Boot application safely?

Behind a feature flag, through an LLM gateway or Spring AI with approved providers, with input/output guardrails, tool authorization in code, observability of tokens and latency, an evaluation set, and cost limits.

How do you test LLM features in Java?

Unit-test prompt assembly and tool logic deterministically; use recorded responses or a local model for integration tests; run evaluation suites (accuracy, faithfulness) as a separate pipeline with thresholds.