LLM Integration: Multiple Providers, Runtime Switching, and Model Comparison

The demo's checklist has one item left: LLM integration — connect with different providers (Anthropic, Google, local models via Ollama), experiment with different models, and compare their performance. Previous rounds built a dependable yardstick (/eval with golden datasets) and a single, interface-based chat model wired into every AI service. This round exploits both: put a registry behind that one interface, and reuse the evaluation harness to compare models head-to-head.

The Problem: One Bean, Many Providers

Until now every AI service (assistant, RAG, agent, few-shot, judge) injected a ChatModel bean built from OpenAI only. Adding providers the obvious way — more @Bean methods, more constructor parameters — would ripple through the whole app. Instead, LangChain4j models all share one interface, ChatModel. So we can keep exactly one ChatModel bean and make it a delegating registry: whichever provider:model is selected is the one that actually answers every service's calls.

LlmProvider

A small enum encodes the four providers and each one's default model:

public enum LlmProvider {
    OPENAI("openai", "gpt-4o-mini"),
    ANTHROPIC("anthropic", "claude-haiku-4-5-20251001"),
    GEMINI("gemini", "gemini-2.5-flash"),
    OLLAMA("ollama", "llama3.2");
    // label() + defaultModelName() + fromLabel(String)
}

ModelRegistry

The registry implements ChatModel, holds the current selection, and lazily builds the real model on first use — cached per provider:model so switching back is free and starting the app never requires a key for a provider you aren't using:

public ChatModel currentChatModel() {
    return models.computeIfAbsent(key(currentProvider, currentModelName),
            ignored -> buildChatModel(currentProvider, currentModelName));
}

private ChatModel buildChatModel(LlmProvider provider, String modelName) {
    return switch (provider) {
        case OPENAI    -> OpenAiChatModel.builder().apiKey(System.getenv("OPENAI_API_KEY")).modelName(modelName).build();
        case ANTHROPIC -> AnthropicChatModel.builder().apiKey(System.getenv("ANTHROPIC_API_KEY")).modelName(modelName).build();
        case GEMINI    -> GoogleAiGeminiChatModel.builder().apiKey(System.getenv("GOOGLE_AI_GEMINI_API_KEY")).modelName(modelName).build();
        case OLLAMA    -> OllamaChatModel.builder().baseUrl(ollamaBaseUrl).modelName(modelName).build();
    };
}

Each provider is one case. The registry also reports which models are available — every provider whose key is present in the environment, plus Ollama, which is local. That list drives the comparison.

Because the registry is a ChatModel and the only ChatModel bean, AiConfig shrinks: the old OpenAI-specific bean disappears and services keep injecting ChatModel unchanged. Switching the registry switches the entire pipeline at once — including the LLM-as-a-judge metric used by /eval.

Runtime Switching in the CLI

/model now covers both kinds of model. Chat switching accepts a provider with or without a model name:

/model
  Chat model     : openai:gpt-4o-mini
  Embedding model: text-embedding-3-small

/model chat anthropic
  Switched chat model to 'anthropic:claude-haiku-4-5-20251001'. Every AI service now uses this model.

/model chat gemini:gemini-2.5-flash

A typo prints the available models with their status (ready, no api key, local), so the CLI doubles as a discovery tool.

Comparing Models: /eval compare

The previous round's EvaluationService evaluates a golden dataset against an answer provider. Cross-model comparison is just running that evaluation once per available model and collecting the averages:

for (String model : modelRegistry.availableModels()) {
    modelRegistry.setModel(model);
    EvaluationReport report = evaluationService.evaluate(dataset, provider);
    rows.add(new ModelScore(model, report.averageScores()));
}

The original selection is restored in a finally block — switching models is a side effect the caller shouldn't be left with. The CLI prints a table:

=== Model comparison: chat ===
Model                                exact  contains  f1  rougeL  embed  judge
openai:gpt-4o-mini                    0.33      0.67 ...
anthropic:claude-haiku-4-5-20251001   0.67      1.00 ...
ollama:llama3.2                       0.33      0.67 ...

This makes the third checklist item ("compare performance and capabilities") concrete: the same golden questions, the same metrics, one row per model.

Testing It Offline

Eight new tests, all offline and deterministic:

• Registry routing — the registry has a test-only constructor pre-populated with FakeChatModels keyed by provider:model. Tests assert the initial selection, provider and provider:model specs both switch correctly, unknown providers are rejected, and doChat delegates to the selected model.

• Comparison end-to-end — a real EvaluationService (with FakeEmbeddingModel for the embed metric and a fake judge) runs against a two-model registry. Tests verify one row per model, all six metric keys in range, that the answer provider observed each model being selected, and that the original selection is restored afterwards.

• Key-free startup — a real constructor test switches to Ollama and builds a model without any API key, proving startup never hard-requires keys.

Design Notes

• One interface, many implementations. Because LangChain4j models are interface-based, "support multiple providers" collapsed into a registry over a single bean type.

• Lazy builds, cached models. No provider is instantiated until used; switching back after a compare costs nothing.

• The judge is a model too. Cross-model comparison keeps its own metric, the LLM-as-a-judge, honest: the judge is the current model, so every row was scored by that model's own judgment.

• Deterministic test constructor. A package-private constructor accepting a pre-populated model map makes the registry testable without network or environment variables; availableModels() is pinned to that map in tests.

Next Steps

The checklist is complete. The demo now spans memory, embeddings, RAG, agents, prompting, integration, tooling, evaluation, advanced features, and multi-provider LLM integration. The natural next experiment is advanced agent orchestration (LangChain4j's agentic module) — with a ChatModel registry that can point it at any provider.

Resources

• LangChain4j Chat Models

• LangChain4j Model Providers

• LLM Integration Issue

Conclusion

The final checklist item turned out to be mostly plumbing: LangChain4j's shared ChatModel interface meant supporting OpenAI, Anthropic, Gemini, and Ollama was a registry of four builder cases behind one bean. Runtime switching gives the demo a way to experiment with models interactively, and /eval compare turns that experimentation into numbers — the same golden datasets and metrics from the evaluation round, now applied per model. The demo began as a single chat command and ends with a switchboard for every major LLM provider plus a local one, all measured by the same yardstick.

The full implementation lives in our GitHub repository.