The ReACT Pattern

ReACT (Reason + Act) is a simple loop: the LLM reasons about what to do, picks a tool, observes the result, and decides whether to continue or answer. Most frameworks hide this loop behind a single chat() call. LangGraph4j exposes it as a first-class state graph.

The graph structure:

__START__ → agent (LLM reasons) → action (tool executes) → agent → ... → __END__

Each node is named, traceable, and inspectable. You can see exactly when the LLM decided to call a tool, what tool it called, what result it got, and how it decided to stop.

What We Built

We integrated LangGraph4j's AgentExecutor into the demo project as a new orchestration endpoint alongside the existing agentic-services patterns (supervisor, chain, parallel, loop, conditional).

@Service
public class ReactAgentService {

    private final CompiledGraph<AgentExecutor.State> compiledGraph;

    public ReactAgentService(ChatModel chatModel, 
                             CalculatorTool calculatorTool,
                             DocumentSearchTool documentSearchTool,
                             WeatherTool weatherTool,
                             EmbeddingStoreStatsTool storeStatsTool) throws GraphStateException {
        StateGraph<AgentExecutor.State> graph = AgentExecutor.builder()
                .chatModel(chatModel)
                .toolsFromObject(calculatorTool, documentSearchTool, weatherTool, storeStatsTool)
                .build();
        this.compiledGraph = graph.compile();
    }

    public ReactResult run(String task) {
        // Stream to capture each graph node transition
        var generator = compiledGraph.stream(Map.of("messages", UserMessage.from(task)));
        for (var item : generator) {
            steps.add(item.node());  // "agent", "action", "agent", ...
        }
        
        // Get final state
        var finalState = compiledGraph.invoke(Map.of("messages", UserMessage.from(task)));
        String answer = finalState.get().finalResponse().orElse("No response");
        
        return new ReactResult(task, answer, steps, allMessages);
    }
}

How It Differs from Built-in Tool Calling

The explicit graph approach means you can:

• Debug reasoning — see exactly why the LLM chose tool A over tool B

• Add guardrails — insert nodes between agent and action

• Build complex patterns — conditional branching, human-in-the-loop (issues #236, #237)

• Monitor performance — track which tools are called, how many loops, where bottlenecks occur

The Pitfalls

LangGraph4j's API has some rough edges:

1. toolsFromObject() takes Object... — not List<Object>. Easy to get wrong.

2. AsyncGenerator doesn't have forEachRemaining() — use a for-each loop instead.

3. AgentExecutor.State messages include ALL intermediate steps — the final state has the complete conversation history including every reasoning step, not just the initial prompt and final answer.

Running It

# CLI
./mvnw spring-boot:run
/react compute 2+2

# REST
curl -X POST http://localhost:8080/api/react \
  -H "Content-Type: application/json" \
  -d '{"message":"What is the weather in Tokyo and what is 15% of 340?"}'

The response includes the full trace:

{
  "task": "What is the weather in Tokyo and what is 15% of 340?",
  "answer": "The weather in Tokyo is 22°C and partly cloudy. 15% of 340 is 51.",
  "steps": ["agent", "action", "agent", "action", "agent"],
  "agentMessages": ["I need to check the weather and calculate 15% of 340.", ...]
}

What's Next

This is the foundation for two more patterns:

• Stateful Pipeline — persist graph state across invocations, resume from checkpoint

• Human-in-the-Loop — pause the graph for human approval before executing sensitive tools

The LangGraph4j integration is the most powerful orchestration pattern we've added — but also the most opinionated. For simple tool use, stick with @AiService. When you need visibility, control, and extensibility, reach for the graph.

The Three Building Blocks

LangChain4j's agentic module provides three workflow patterns that can be nested inside each other:

Parallel (parallelBuilder)

Runs sub-agents concurrently, then merges results via an output() function:

UntypedAgent parallelResearch = AgenticServices.<String>parallelBuilder()
        .subAgents(researchAgent1, researchAgent2)
        .outputKey("research")
        .output(scope -> {
            String r1 = scope.readState("research1", "");
            String r2 = scope.readState("research2", "");
            return r1 + "\n\n" + r2;
        })
        .build();

Loop (loopBuilder)

Runs sub-agents repeatedly until an exit condition is met:

UntypedAgent refinementLoop = AgenticServices.<String>loopBuilder()
        .subAgents(qualityScorer, improveAgent)
        .maxIterations(3)
        .exitCondition(scope -> scope.readState("score", 0.0) >= 0.8)
        .build();

Conditional (conditionalBuilder)

Routes to different sub-agents based on a predicate:

UntypedAgent conditionalFormatter = AgenticServices.<String>conditionalBuilder()
        .subAgents(
                scope -> "technical".equals(scope.readState("category", "")),
                technicalFormat)
        .subAgents(
                scope -> !"technical".equals(scope.readState("category", "")),
                generalFormat)
        .build();

The Demo Pipeline

The workflow generates a blog post with four phases:

flowchart TD
    subgraph P["Phase 1: Parallel Research"]
        R1[ResearchAgent1<br/>research1]
        R2[ResearchAgent2<br/>research2]
    end

    subgraph L["Phase 3: Refinement Loop"]
        QS[QualityScorerAgent<br/>score]
        IA[ImproveAgent<br/>draft]
    end

    P --> D[Phase 2: DraftAgent<br/>draft]
    D --> L
    L --> CA[Phase 4: CategoryAgent<br/>category]
    CA -->|technical| TF[TechnicalFormatAgent<br/>formatted]
    CA -->|general| GF[GeneralFormatAgent<br/>formatted]

All four patterns compose in a single sequence:

this.pipeline = AgenticServices.sequenceBuilder()
        .subAgents(parallelResearch, draftAgent, refinementLoop, categoryAgent)
        .subAgents(conditionalFormatter)
        .outputKey("formatted")
        .build();

Gotchas

Three things worth noting:

1. Builders must call .build(). parallelBuilder(), loopBuilder(), and conditionalBuilder() return builder objects. You must call .build() to get the UntypedAgent that can be passed to sequenceBuilder().subAgents(). Forgetting .build() causes a cryptic "No agent method found" error.

2. Parallel output() merges scope keys. The output() function on parallelBuilder reads sub-agent outputs from the scope and combines them. Set outputKey() to write the merged result back to the scope.

3. Loop exitCondition is checked after each iteration. The predicate receives the live AgenticScope. The loop runs sub-agents sequentially within each iteration — so if you have [scorer, improver], iteration 1 runs scorer then improver, and iteration 2 runs scorer again (checking the exit condition).

What We Learned

The three workflow patterns cover the common orchestration cases:

• Parallel for independent work that can run concurrently (research, brainstorming)

• Loop for iterative refinement (quality scoring, style review)

• Conditional for branching logic (formatting, routing)

• Sequence for chaining them together

The key insight is that all three are composable. A loop can wrap a parallel workflow. A conditional can branch to a parallel or loop sub-graph. The sequenceBuilder orchestrates them in order.

This completes the orchestration feature set for the demo: crew (supervisor delegation), chain (sequential pipeline), graph (GOAP planning), and workflow (parallel/loop/conditional composition).

Next up: The demo now covers all core orchestration patterns. The remaining open idea is parallel agent execution — running multiple agents concurrently and merging their results, which is partially covered by the parallel pattern here.

What Is GOAP?

GOAP stands for Goal-Oriented Action Planning. In LangChain4j, the GoalOrientedPlanner (from the langchain4j-agentic-patterns module) analyzes the dependency graph of your agents and uses A* search to find the shortest path from the current scope state to the goal.

Each agent declares:

• Preconditions — what it needs from the scope (via @V("key") parameters)

• Postconditions — what it writes to the scope (via .outputKey())

The planner builds a directed graph from these declarations, then finds the shortest path that satisfies all dependencies. No LLM call is needed for planning — it's pure graph search.

The Pipeline

The demo uses a personalized blog post generation pipeline with six agents:

The goal is writeup. Given an initial scope containing prompt, the planner computes the shortest path:

prompt → profile → topic → outline → draft → edited → writeup

flowchart LR
    P["prompt"] --> EA["ExtractProfile<br/>profile"]
    EA --> TS["TopicSuggestion<br/>topic"]
    TS --> TO["TopicOutline<br/>outline"]
    TS --> TD["TopicDraft<br/>draft"]
    TO --> TD
    TD --> TE["TopicEditor<br/>edited"]
    TE --> TW["TopicWriteup<br/>writeup"]

The Code

Each agent is a plain Java interface:

@SystemMessage("You are a profile extractor...")
@UserMessage("Extract a short profile from the following prompt...{{prompt}}")
public interface ExtractProfileAgent {
    @Agent(outputKey = "profile", description = "Extracts a user profile from a prompt")
    String extractProfile(@V("prompt") String prompt);
}

The TopicWriteupAgent reads four scope keys and writes the final output:

@SystemMessage("You are a blog post formatter...")
@UserMessage("""
        Create a personalized blog post using the following:
        - User profile: {{profile}}
        - Topic: {{topic}}
        - Outline: {{outline}}
        - Edited content: {{edited}}
        """)
public interface TopicWriteupAgent {
    @Agent(outputKey = "writeup", description = "Formats a personalized blog post")
    String createWriteup(@V("profile") String profile,
                         @V("topic") String topic,
                         @V("outline") String outline,
                         @V("edited") String edited);
}

The pipeline is assembled in GraphOfAgentsService:

this.pipeline = AgenticServices.plannerBuilder()
        .subAgents(extractProfile, topicSuggestion, outline, draft, editor, writeup)
        .outputKey("writeup")
        .planner(GoalOrientedPlanner::new)
        .build();

That's it. The planner does the rest.

How the Planner Works

When pipeline.invoke(Map.of("prompt", "...")) is called:

1. Graph construction. The GoalOrientedSearchGraph iterates all sub-agents, extracts their @V parameter names (preconditions) and .outputKey() values (postconditions), and builds a directed graph.

2. Path search. The DependencyGraphSearch (A* algorithm) starts from the initial scope state ({prompt}) and finds the shortest path to the goal node (writeup).

3. Execution. The planner invokes each agent in the computed order, writing results to the scope as it goes. Each agent's @V parameters are automatically populated from the scope.

4. Result. The final output is the value of the goal key (writeup) from the scope.

The key insight: the planner is algorithmic, not LLM-driven. No LLM call is needed to decide which agent runs next. The graph search is deterministic and fast.

Gotchas

Three things worth noting:

1. langchain4j-agentic-patterns is a separate module. It has its own -betaNN version and must be added to pom.xml alongside langchain4j-agentic. The GOAP classes live in dev.langchain4j.agentic.patterns.goap.

2. plannerBuilder() has no chatModel() method. The chatModel is set on each individual agent builder, not on the planner builder. The planner is purely an orchestration layer — it doesn't invoke any LLM itself.

3. Adding a new agent is declarative. Just add a new interface with @V parameters and outputKey, build it with agentBuilder(), and add it to the plannerBuilder().subAgents(...) list. The planner automatically discovers its dependencies and recalculates the path.

What We Learned

The GOAP pattern completes the orchestration trilogy: crew (supervisor delegation), chain (sequential pipeline), and graph (goal-oriented planning). The key difference is that the graph pattern doesn't require you to specify the execution order — you just declare what each agent needs and produces, and the planner figures out the rest.

This is powerful when the pipeline might vary depending on what's already available. If the initial scope already contains a topic, the planner automatically skips ExtractProfileAgent and TopicSuggestionAgent. The same agents work regardless of which scope keys are pre-populated.

The langchain4j-agentic-patterns module also includes BDI, Blackboard, Debate, P2P, and Voting patterns — all built on the same Planner abstraction. Each pattern is composable: a GOAP sub-graph can wrap a loop, a conditional, or a parallel workflow as one of its agents.

Next up: The demo now has all three core orchestration patterns. The remaining open idea is parallel agent execution — running multiple agents concurrently and merging their results.

What Is a Chain of Agents?

LangChain4j's AgenticServices.sequenceBuilder() creates an UntypedAgent from a list of sub-agents that execute in order. Each sub-agent reads its input from the shared scope, writes its result to a named key, and the next agent picks up where the last one left off. There is no routing decision, no supervisor choosing which worker to call — it is a fixed, linear pipeline.

This is useful when the stages are known in advance and each one depends on the previous stage's output. The demo uses a blog-post generation pipeline:

1. OutlineAgent — takes a topic, produces a structured outline (outputKey = "outline")

2. DraftAgent — takes the outline, writes a full draft (outputKey = "draft")

3. EditorAgent — takes the draft, edits for clarity and flow (outputKey = "edited")

4. FormatAgent — takes the edited text, formats into a publish-ready Markdown post (outputKey = "formatted")

flowchart LR
    T["topic input"] --> OA["OutlineAgent<br/>outputKey=outline"]
    OA --> DA["DraftAgent<br/>outputKey=draft"]
    DA --> EA["EditorAgent<br/>outputKey=edited"]
    EA --> FA["FormatAgent<br/>outputKey=formatted"]
    FA --> OUT["formatted blog post"]

The Code

Each sub-agent is a plain Java interface with @SystemMessage, @UserMessage, and @Agent(outputKey = "..."):

@SystemMessage("You are a blog post outline specialist. Create a clear, structured outline "
        + "with a title, introduction, 3-5 main sections, and a conclusion.")
@UserMessage("Create a blog post outline for the following topic.\nTopic: {{topic}}")
public interface OutlineAgent {
    @Agent(outputKey = "outline", description = "Creates a structured blog post outline")
    String createOutline(@V("topic") String topic);
}

DraftAgent, EditorAgent, and FormatAgent follow the same pattern, each with its own outputKey and system message tailored to its role.

The pipeline is assembled in ChainOfAgentsService:

OutlineAgent outlineAgent = AgenticServices.agentBuilder(OutlineAgent.class)
        .chatModel(chatModel).build();
DraftAgent draftAgent = AgenticServices.agentBuilder(DraftAgent.class)
        .chatModel(chatModel).build();
EditorAgent editorAgent = AgenticServices.agentBuilder(EditorAgent.class)
        .chatModel(chatModel).build();
FormatAgent formatAgent = AgenticServices.agentBuilder(FormatAgent.class)
        .chatModel(chatModel).build();

this.pipeline = AgenticServices.sequenceBuilder()
        .subAgents(outlineAgent, draftAgent, editorAgent, formatAgent)
        .outputKey("formatted")
        .build();

Two details:

• Each sub-agent is built independently with its own AgenticServices.agentBuilder() call, then composed via sequenceBuilder(). This is different from the crew, where the supervisor and sub-agents are built together.

• outputKey("formatted") tells the pipeline which scope key becomes the top-level return value of pipeline.invoke().

To get the full trace (all intermediate outputs), use invokeWithAgenticScope():

ResultWithAgenticScope<String> result =
        pipeline.invokeWithAgenticScope(Map.of("topic", topic));
String formatted = result.result();
String outline = result.agenticScope().readState("outline", (String) null);
String draft = result.agenticScope().readState("draft", (String) null);

The REST endpoint (POST /api/chain) and CLI command (/chain <topic>) both return the full trace, not just the final output.

Gotchas

Three things tripped us up during implementation:

1. @SystemMessage and @UserMessage go on the method, not the interface. Placing them at the interface level produces a compile error: annotation interface not applicable to this kind of declaration. This is the same rule as for crew sub-agents, but it is easy to forget when the interface has only one method.

2. UntypedAgent.invoke() returns Object, not Map<String, Object>. The return value is the value of the configured outputKey, not the full scope map. To access intermediate outputs, you must use invokeWithAgenticScope() and agenticScope().readState(key).

3. Scope keys must be unique across all sub-agents. Each agent's outputKey writes to a different key in the shared scope. If two agents share a key, the second one silently overwrites the first.

Testing

The ChainOfAgentsServiceTest uses a ScriptedSequenceChatModel — a fake ChatModel that returns canned responses in order, one per agent invocation. This makes the test fully offline with no API calls:

ScriptedSequenceChatModel chatModel = new ScriptedSequenceChatModel(
        List.of(OUTLINE, DRAFT, EDITED, FORMATTED));
ChainOfAgentsService service = new ChainOfAgentsService(chatModel);

ChainPipelineResult result = service.runWithTrace("Test Topic");

assertThat(result.outline()).isEqualTo(OUTLINE);
assertThat(result.draft()).isEqualTo(DRAFT);
assertThat(result.edited()).isEqualTo(EDITED);
assertThat(result.formatted()).isEqualTo(FORMATTED);

What We Learned

The sequenceBuilder() fills a real gap between "one agent, one call" and "supervisor with routing." When the pipeline is fixed — outline, draft, edit, format — there is no need for a supervisor to decide which agent runs next. The shared AgenticScope is the glue: each agent writes its result, the next agent reads it, and the final outputKey gives you the finished product.

This also reinforces the value of the ModelRegistry abstraction. Every agent in the chain shares the same ChatModel bean, so switching providers at runtime switches the entire pipeline in one move. The registry keeps paying for itself.

Next up: The demo now has a supervisor-based crew (parallel delegation) and a chain-of-agents pipeline (sequential delegation). The remaining open idea is parallel agent execution — running multiple agents concurrently and merging their results.

The Two Experiments

1. The crew. The agentic module (currently 1.18.0-beta28) provides AgenticServices.agentBuilder() for standalone agents and AgenticServices.supervisorBuilder() for a supervisor that delegates to sub-agents. It isn't in the BOM, so pom.xml pins its own -betaNN version explicitly; it depends on langchain4j 1.18.0, which the BOM's 1.18.1 overrides.

2. Streaming function calling. AiServices already supports streaming services — a TokenStream returned from the method, tokens arriving via onPartialResponse. Binding @Tools to a streaming service is no different from binding them to a blocking one. The two compose: StreamingAgent.chat(...) returns a TokenStream, and the tool is still available to the model mid-generation.

CrewService: A Supervisor With Three Workers

CrewService wires a supervisor to three sub-agents, each bound to one of the demo's existing @Tools — the calculator, the weather lookup, and the document research tool:

CrewTaskAgent calculatorAgent = buildAgent(
        "Calculator", "Useful for arithmetic and any kind of math. Delegate calculations here.",
        chatModel, calculatorTool);
CrewTaskAgent weatherAgent = buildAgent(
        "Weather", "Useful for current weather in known cities. Delegate weather questions here.",
        chatModel, weatherTool);
CrewTaskAgent researchAgent = buildAgent(
        "Research", "Useful for questions about the indexed documents. Delegate document questions here.",
        chatModel, documentSearchTool);

this.supervisor = AgenticServices.supervisorBuilder()
        .name("Crew")
        .supervisorContext("You are the crew supervisor. Decide which agent is best suited for the task ...")
        .chatModel(chatModel)
        .chatMemoryProvider(memoryProvider)
        .subAgents(calculatorAgent, weatherAgent, researchAgent)
        .responseStrategy(SupervisorResponseStrategy.LAST)
        .maxAgentsInvocations(10)
        .build();

Two details worth highlighting:

• Every agent shares one ChatModel — the ModelRegistry from the LLM-integration round. A sub-agent is just a ChatModel consumer, so switching providers with /model chat switches the whole crew in one move. The registry abstraction keeps paying for itself.

• Sub-agents must be typed. The supervisor hands a sub-agent an arguments map whose keys are matched to the sub-agent's parameters by name. That only works when the sub-agent declares its input as a @V parameter on a typed method with a @UserMessage template:

public interface CrewTaskAgent {

    @SystemMessage("You are a specialist agent. Complete the task delegated to you using the tools "
            + "available. Return the final answer and nothing else.")
    @UserMessage("You have been delegated the following task. Use the tools available to complete it.\n"
            + "Delegated task: {{task}}")
    @Agent
    String run(@V("task") String task);
}

It was tempting to build sub-agents as untyped AgenticServices.agentBuilder() with a userMessageProvider extracting the task key. That fails silently: the provider never receives the delegated map (it gets the default memory id instead), so the worker ends up answering the literal string "default". The typed contract makes the hand-off explicit — the supervisor's {"task": ...} argument becomes the run(...) argument, and the template builds the worker's prompt from it.

How the supervisor actually works

The supervisor does not hand the model a JSON-tools list of its sub-agents. Its SupervisorPlanner asks the model for a decision in a specific shape — an AgentInvocation POJO with agentName and arguments — and the runtime invokes the matching sub-agent:

• Model returns {"agentName": "Weather", "arguments": {...}} → the Weather sub-agent runs, calls the model itself (and, if needed, its getWeather tool).

• Model returns {"agentName": "done"} → the planner stops; with SupervisorResponseStrategy.LAST, the final answer is the last sub-agent's output.

That protocol is what made the tests deterministic (more below).

Streaming Function Calling

StreamingAgent is a one-method AI service:

public interface StreamingAgent {
    TokenStream chat(@MemoryId String memoryId, @UserMessage String message);
}

Its bean binds the existing OpenAiStreamingChatModel and the CalculatorTool:

AiServices.builder(StreamingAgent.class)
        .streamingChatModel(streamingChatModel)
        .chatMemoryProvider(createChatMemoryProvider(...))
        .tools(calculatorTool)
        .build();

The /stream command prints tokens as they arrive; the model can still decide to call calculate before producing its final text.

The CLI Commands

Two new commands complete the surface:

/crew <task>     Execute a task with the agentic supervisor crew
/stream <task>   Stream a task with streaming function calling

TokenStream tokenStream = streamingAgent.chat(memoryId, task);
tokenStream
        .onPartialResponse(System.out::print)
        .onCompleteResponse(response -> System.out.println())
        .onError(error -> System.out.println("Stream error: " + error.getMessage()))
        .start();

Testing It Offline

Three new tests, all offline and deterministic:

• Crew delegation protocol. A ScriptedSupervisorChatModel returns the three model replies in order: {"agentName":"Weather",...} (supervisor delegates), then a plain sub-agent answer, then {"agentName":"done"} (supervisor wraps up). The test asserts the returned answer is the sub-agent's text and that the supervisor made at least three calls.

• Delegated task delivery. The recorded second request — the Weather sub-agent's call — must contain the delegated task ("weather report") in its user message, proving the supervisor's arguments really reach the worker's prompt.

• Sub-agent tool exposure. The same recorded second request must carry the getWeather tool specification, proving the @Tool really is bound inside the crew.

• Streaming tokens + tools. A FakeStreamingChatModel (extended with lastRequest()) emits three tokens; the test collects them through onPartialResponse, asserts the full text on onCompleteResponse, and verifies the request's tool specifications contain calculate.

Full suite: 106 tests, all green.

Design Notes

• The supervisor is a small protocol, not a tool call. Understanding that the model's output is parsed into an AgentInvocation was the key to both the implementation and the tests. Get the shape wrong and you get OutputParsingException: Failed to parse ... into AgentInvocation.

• Arguments are matched to sub-agent parameters by name. That is why the worker must be typed with @V("task"); a generic untyped userMessageProvider never receives the delegated arguments. The regression test asserting the delegated text in the worker's prompt would have caught this earlier.

• LAST keeps the loop short. SupervisorResponseStrategy.LAST returns the delegated result directly instead of scoring or summarizing it, which also makes the test's call sequence stable.

• Shared model = coherent crew. Because sub-agents take a ChatModel, the whole crew inherits runtime provider switching for free.

Next Steps

The checklist and the "Future Experiments" list are now empty: memory, embeddings, RAG, agents, prompting, integration, tooling, evaluation, advanced features, multi-provider LLM integration, and now agentic orchestration plus streaming function calling. A natural follow-up is giving the crew more sub-agents (e.g., an image or audio worker) and a plan-then-execute workflow — the agentic module keeps growing.

Resources

• LangChain4j agentic module

• LangChain4j Streaming

• Advanced Orchestration Issue

Conclusion

The last item on the board turned out to be the most mechanically involved. The agentic module moves orchestration out of your prompt and into typed primitives: agents as builders, delegation as a parsed decision, and a supervisor loop that knows when to stop. Combined with streaming function calling, the demo now shows both ends of the spectrum — a single streaming tool-calling round and a multi-agent crew that plans its own hand-off. And because everything shares the ModelRegistry, one command switches the entire crew to a different provider or model.

The full implementation lives in our GitHub repository.

The Problem: One Bean, Many Providers

Until now every AI service (assistant, RAG, agent, few-shot, judge) injected a ChatModel bean built from OpenAI only. Adding providers the obvious way — more @Bean methods, more constructor parameters — would ripple through the whole app. Instead, LangChain4j models all share one interface, ChatModel. So we can keep exactly one ChatModel bean and make it a delegating registry: whichever provider:model is selected is the one that actually answers every service's calls.

LlmProvider

A small enum encodes the four providers and each one's default model:

public enum LlmProvider {
    OPENAI("openai", "gpt-4o-mini"),
    ANTHROPIC("anthropic", "claude-haiku-4-5-20251001"),
    GEMINI("gemini", "gemini-2.5-flash"),
    OLLAMA("ollama", "llama3.2");
    // label() + defaultModelName() + fromLabel(String)
}

ModelRegistry

The registry implements ChatModel, holds the current selection, and lazily builds the real model on first use — cached per provider:model so switching back is free and starting the app never requires a key for a provider you aren't using:

public ChatModel currentChatModel() {
    return models.computeIfAbsent(key(currentProvider, currentModelName),
            ignored -> buildChatModel(currentProvider, currentModelName));
}

private ChatModel buildChatModel(LlmProvider provider, String modelName) {
    return switch (provider) {
        case OPENAI    -> OpenAiChatModel.builder().apiKey(System.getenv("OPENAI_API_KEY")).modelName(modelName).build();
        case ANTHROPIC -> AnthropicChatModel.builder().apiKey(System.getenv("ANTHROPIC_API_KEY")).modelName(modelName).build();
        case GEMINI    -> GoogleAiGeminiChatModel.builder().apiKey(System.getenv("GOOGLE_AI_GEMINI_API_KEY")).modelName(modelName).build();
        case OLLAMA    -> OllamaChatModel.builder().baseUrl(ollamaBaseUrl).modelName(modelName).build();
    };
}

Each provider is one case. The registry also reports which models are available — every provider whose key is present in the environment, plus Ollama, which is local. That list drives the comparison.

Because the registry is a ChatModel and the only ChatModel bean, AiConfig shrinks: the old OpenAI-specific bean disappears and services keep injecting ChatModel unchanged. Switching the registry switches the entire pipeline at once — including the LLM-as-a-judge metric used by /eval.

Runtime Switching in the CLI

/model now covers both kinds of model. Chat switching accepts a provider with or without a model name:

/model
  Chat model     : openai:gpt-4o-mini
  Embedding model: text-embedding-3-small

/model chat anthropic
  Switched chat model to 'anthropic:claude-haiku-4-5-20251001'. Every AI service now uses this model.

/model chat gemini:gemini-2.5-flash

A typo prints the available models with their status (ready, no api key, local), so the CLI doubles as a discovery tool.

Comparing Models: /eval compare

The previous round's EvaluationService evaluates a golden dataset against an answer provider. Cross-model comparison is just running that evaluation once per available model and collecting the averages:

for (String model : modelRegistry.availableModels()) {
    modelRegistry.setModel(model);
    EvaluationReport report = evaluationService.evaluate(dataset, provider);
    rows.add(new ModelScore(model, report.averageScores()));
}

The original selection is restored in a finally block — switching models is a side effect the caller shouldn't be left with. The CLI prints a table:

=== Model comparison: chat ===
Model                                exact  contains  f1  rougeL  embed  judge
openai:gpt-4o-mini                    0.33      0.67 ...
anthropic:claude-haiku-4-5-20251001   0.67      1.00 ...
ollama:llama3.2                       0.33      0.67 ...

This makes the third checklist item ("compare performance and capabilities") concrete: the same golden questions, the same metrics, one row per model.

Testing It Offline

Eight new tests, all offline and deterministic:

• Registry routing — the registry has a test-only constructor pre-populated with FakeChatModels keyed by provider:model. Tests assert the initial selection, provider and provider:model specs both switch correctly, unknown providers are rejected, and doChat delegates to the selected model.

• Comparison end-to-end — a real EvaluationService (with FakeEmbeddingModel for the embed metric and a fake judge) runs against a two-model registry. Tests verify one row per model, all six metric keys in range, that the answer provider observed each model being selected, and that the original selection is restored afterwards.

• Key-free startup — a real constructor test switches to Ollama and builds a model without any API key, proving startup never hard-requires keys.

Design Notes

• One interface, many implementations. Because LangChain4j models are interface-based, "support multiple providers" collapsed into a registry over a single bean type.

• Lazy builds, cached models. No provider is instantiated until used; switching back after a compare costs nothing.

• The judge is a model too. Cross-model comparison keeps its own metric, the LLM-as-a-judge, honest: the judge is the current model, so every row was scored by that model's own judgment.

• Deterministic test constructor. A package-private constructor accepting a pre-populated model map makes the registry testable without network or environment variables; availableModels() is pinned to that map in tests.

Next Steps

The checklist is complete. The demo now spans memory, embeddings, RAG, agents, prompting, integration, tooling, evaluation, advanced features, and multi-provider LLM integration. The natural next experiment is advanced agent orchestration (LangChain4j's agentic module) — with a ChatModel registry that can point it at any provider.

Resources

• LangChain4j Chat Models

• LangChain4j Model Providers

• LLM Integration Issue

Conclusion

The final checklist item turned out to be mostly plumbing: LangChain4j's shared ChatModel interface meant supporting OpenAI, Anthropic, Gemini, and Ollama was a registry of four builder cases behind one bean. Runtime switching gives the demo a way to experiment with models interactively, and /eval compare turns that experimentation into numbers — the same golden datasets and metrics from the evaluation round, now applied per model. The demo began as a single chat command and ends with a switchboard for every major LLM provider plus a local one, all measured by the same yardstick.

The full implementation lives in our GitHub repository.

Multi-Modal

"Multi-modal" here means three separate capabilities, each backed by a different model interface in LangChain4j.

Vision: images into the chat model

A multimodal chat model (GPT-4o-mini) accepts image content alongside text. VisionService builds a UserMessage that mixes a TextContent (the question) with an ImageContent:

UserMessage.from(TextContent.from(question), ImageContent.from(URI.create(imageUrl)))

The image can come from a public URL or, more usefully, as raw bytes — the service base64-encodes them and passes a mime type, which is how an uploaded file reaches the model:

ImageContent.from(Base64.getEncoder().encodeToString(imageData), mimeType)

POST /api/describe exposes this over HTTP ({imageUrl | imageData, mimeType, question}), and /describe <url> [question] works from the CLI.

Image generation

ImageGenerationService wraps an ImageModel — OpenAI's gpt-image-1 by default, configurable via app.image.model-name. The returned Image carries a URL, or base64 data plus a mime type, or a revisedPrompt. The CLI /generate <prompt> prints whichever form came back.

Speech-to-text

SpeechToTextService wraps an AudioTranscriptionModel (whisper-1). It wraps raw audio bytes into an Audio value and calls transcribeToText:

Audio audio = Audio.builder()
        .base64Data(Base64.getEncoder().encodeToString(audioData))
        .mimeType(mimeType)
        .build();
return transcriptionModel.transcribeToText(audio);

CLI: /transcribe <file>. The wav/mp3/ogg/m4a extension is mapped to a mime type so the model knows what it's parsing.

Function Calling and Custom Tools

The agent from an earlier round already calls @Tool methods. This round pushes further on what a tool can be.

Structured tool parameters

WeatherTool.getWeather takes a WeatherRequest record instead of a handful of scalars:

@Tool("Gets the current temperature for a city, returning it in the requested unit")
public String getWeather(WeatherRequest request) { ... }

@Description("Weather request parameters")
public record WeatherRequest(String city, TemperatureUnit unit) { }

public enum TemperatureUnit { CELSIUS, FAHRENHEIT }

LangChain4j derives a nested object schema from the record — city as a string, unit as an enum — so the model can fill in both fields in one call. The weather data is fake and deterministic, so the tool works offline. One subtlety surfaced in testing: for a POJO parameter named request, the model's arguments must be {"request": {"city": "...", "unit": "..."}}, keyed by the parameter name, not the bare object.

Conversation-scoped tool state

NoteTool demonstrates a tool with state that is aware of which conversation it is running in, via @ToolMemoryId:

@Tool("Saves a note for the current conversation")
public String saveNote(@ToolMemoryId String memoryId, @P("The note text to save") String note) { ... }

LangChain4j injects the current conversation's memory id into the call, so notes saved in one conversation never leak into another — a real pattern for memory-adjacent tools.

Dynamic tool selection with a ToolProvider

The static agent registers its tools at build time with .tools(...). The new DynamicToolProvider instead decides per request which tools are available, using ToolProviderRequest to inspect the user message:

@Override
public ToolProviderResult provideTools(ToolProviderRequest request) {
    List<AiServiceTool> tools = new ArrayList<>(ToolService.findTools(calculatorTool));
    tools.addAll(ToolService.findTools(noteTool));
    if (request.userMessage().singleText().toLowerCase().contains("weather")) {
        tools.addAll(ToolService.findTools(weatherTool));
    }
    return ToolProviderResult.builder().addAll(tools).build();
}

The calculator and note tools are always exposed; the weather tool only appears when the task is actually about the weather. That keeps the model's function-call surface — and the tokens describing it — as small as possible. AiServices.builder(DynamicAgent.class).toolProvider(provider) wires it up, and /dynamic <task> exercises it from the CLI.

Structured Output at the Model Level

The demo already had structured output through AI Service return types (MovieReview, enums, List<String>), where LangChain4j asks for JSON in the prompt and parses the reply. This round adds the model-level approach: attach the JSON schema derived from the record to the request as a response format, constraining the model to emit exactly that shape:

JsonSchema schema = JsonSchemas.jsonSchemaFrom(MovieReview.class)
        .orElseThrow(...);

ChatRequest request = ChatRequest.builder()
        .messages(List.of(SystemMessage.from(SYSTEM_PROMPT), UserMessage.from(text)))
        .parameters(ChatRequestParameters.builder()
                .responseFormat(schema)   // model is constrained to this schema
                .build())
        .build();

/schema <text> runs this from the CLI. The guarantee is stronger than prompt-based extraction: with JSON mode the model's output is structurally enforced rather than merely requested.

Testing It All Offline

The suite grew to 92 tests, all offline:

• Vision — a FakeChatModel captures the built ChatRequest; tests assert the user message contains exactly one TextContent and one ImageContent, and that the base64 variant carries the right mime type.

• Image generation / STT — dedicated fakes (FakeImageModel, FakeAudioTranscriptionModel) capture the prompt and the Audio payload respectively.

• Custom tools — direct unit tests for WeatherTool (unit conversion, unknown city) and NoteTool (notes scoped per conversation).

• Function calling end-to-end — DynamicToolProviderTest builds a real DynamicAgent over AiServices with a fake chat model that returns a ToolExecutionRequest on its first call and a plain answer on its second. The test watches the tool-specification list on each round: the weather tool is present only when the task mentions weather, the tool actually executes, and its result flows back into the second model call.

• Structured output — asserts the request's responseFormat().jsonSchema() is named MovieReview and that the model's JSON reply parses into the record.

The agent-loop test deserves emphasis: it exercises the real AiServices machinery — schema generation, tool dispatch, the round-trip of results back into the model — with zero network calls, purely by scripting what the model would say.

Design Notes

• Fake models everywhere. No capability ships without an offline test proving the request wiring is correct.

• Runtime configurability. Image and STT models are beans you can swap by property, matching the existing chat/embedding model pattern.

• Dynamic > static where it matters. Exposing a tool per request beats registering every tool for every task — smaller prompts, fewer wrong calls.

• Nested tool schemas. A record parameter is the clean way to give a tool rich, typed inputs; just remember the arguments are keyed by the parameter name.

Next Steps

The last checklist item is LLM integration: wiring alternative providers (Anthropic, Google, Ollama) and comparing models. The /eval harness from the previous round is exactly the yardstick that comparison needs.

Resources

• LangChain4j Tools Documentation

• LangChain4j Multi-Modality

• Advanced Features Issue

Conclusion

The advanced-features round turned the demo from a text-only assistant into one that sees images, paints them, listens, calls tools with structured parameters, and emits schema-constrained JSON. Two ideas carried the round: fakes make even a tool-calling agent loop testable offline, and the interface-based model abstraction in LangChain4j (chat, image, audio) means each new modality was a small service rather than a rewrite.

The full implementation lives in our GitHub repository.

The Setup: Golden Datasets

Evaluation needs a ground truth. Each GoldenDataset is a named list of (question, expectedAnswer) pairs, and we bundle three, one per capability:

public record GoldenDataset(String name, List<GoldenQuestion> goldenQuestions) {
    public static GoldenDataset rag() {
        return new GoldenDataset("rag", List.of(
                new GoldenQuestion(
                        "How does LangChain4j keep conversation memory?",
                        "LangChain4j offers MessageWindowChatMemory and TokenWindowChatMemory."),
                new GoldenQuestion(
                        "How does semantic search find similar texts?",
                        "Semantic search embeds the query and finds stored vectors with the highest cosine similarity."),
                new GoldenQuestion(
                        "What is LangChain4j?",
                        "LangChain4j is a Java framework that simplifies building applications with LLMs.")
        ));
    }
    // sentiment() and chat() follow the same shape
}

The RAG dataset's expected answers are written against the bundled sample-data/ documents, so the questions are genuinely answerable by retrieval alone. Each dataset is deliberately small — three samples — because small and fast means we can run it on every test run and in CI.

The Metrics

Metric is a one-method interface returning a score in [0, 1]:

public interface Metric {
    String name();
    double evaluate(String question, String expected, String actual);
}

The Metrics factory builds six of them:

The first four are pure string math. embed reuses the app's embedding service, so "v1.0 vectors with highest cosine similarity" scores high against "embeds the query and finds stored vectors with the highest cosine similarity" even though only some words match — that's exactly the semantic tolerance you want from a search system.

LLM-as-a-judge

The judge metric is the only one that calls a model. It sends a strict system prompt plus the question, expected answer, and produced answer, then asks for a single integer 0–5:

ChatResponse response = judge.chat(ChatRequest.builder()
        .messages(List.of(
                SystemMessage.from(JUDGE_SYSTEM_PROMPT),
                UserMessage.from("Question: " + question + "\n"
                        + "Expected answer: " + expected + "\n"
                        + "Produced answer: " + actual)))
        .build());
return parseJudgeScore(response.aiMessage().text()) / 5.0;

LLMs are notoriously bad at formatting, so parseJudgeScore doesn't trust the reply: it extracts the first run of digits with a regex and clamps into 0–5. "Score: 5/5", "5", and "I'd say 5!" all parse to 1.0. A reply with no digits — say the model went on a tangent — scores 0.0 rather than crashing the run.

EvaluationService: Score a Provider

EvaluationService ties it together. It takes an AnswerProvider — a plain String answer(String question) function — runs the dataset through it, and scores every answer with every metric:

public EvaluationReport evaluate(GoldenDataset dataset, AnswerProvider provider, List<Metric> metrics) {
    // for each golden question:
    //   actual = provider.answer(question)
    //   scores = each metric.evaluate(question, expected, actual)
    // averages = per-metric mean, rounded to 2 decimals
}

The abstraction is what makes evaluation reusable. The /eval command binds each dataset to its real system with a one-line lambda:

case "rag" -> {
    if (searchService.storeSize() == 0) { /* nudge user to /index first */ }
    provider = question -> qaService.ask(memoryId, question);   // full RAG pipeline
}
case "chat" -> provider = question -> assistant.chat(memoryId, question);
case "sentiment" -> provider = question -> fewShotAssistant.classify(question).name();

The default metric set — exact, contains, f1, rougeL, embed, judge — is wired in EvaluationService.defaultMetrics().

The Report

/eval prints a per-question breakdown and the averages:

=== Evaluation: sentiment ===
Metrics: exact, contains, f1, rougeL, embed, judge

[1] I absolutely loved this movie!
    Expected: POSITIVE
    Actual  : POSITIVE
    Scores  : exact=1.00  contains=1.00  f1=1.00  rougeL=1.00  embed=1.00  judge=1.00

[2] The service was okay, nothing special.
    Expected: NEUTRAL
    Actual  : NEUTRAL
    Scores  : exact=1.00  contains=1.00  f1=1.00  rougeL=1.00  embed=1.00  judge=1.00

Average: exact=1.00  contains=1.00  f1=1.00  rougeL=1.00  embed=1.00  judge=1.00

The multi-metric view is the point: exact and contains are brittle, f1/rougeL reward partial matches, embed tolerates rephrasing, and judge captures quality the string metrics can't. When one model replaces another, the averages shift in ways that tell you which failure mode changed — verbatim accuracy versus factual recall versus phrasing.

Offline Testing: The Real Pipeline, Fake Models

The golden datasets double as tests. EvaluationServiceTest runs the full RAG stack with real components — AiServices.builder creating the QaAssistant proxy, MessageWindowChatMemory, a DefaultRetrievalAugmentor with the real SemanticSearchContentRetriever, a temp document indexed into a real InMemoryEmbeddingStore — and only the two models are fakes:

QaAssistant qaAssistant = AiServices.builder(QaAssistant.class)
        .chatModel(fakeChatModel)
        .chatMemory(MessageWindowChatMemory.withMaxMessages(10))
        .retrievalAugmentor(DefaultRetrievalAugmentor.builder()
                .contentRetriever(new SemanticSearchContentRetriever(searchService))
                .build())
        .build();

EvaluationReport report = evaluationService.evaluate(
        GoldenDataset.rag(), question -> qaService.ask("rag", question));

The fake chat model returns a canned answer for every prompt, so every RAG question scores exact=1.0 — the test asserts the harness behaves end-to-end rather than testing the LLM (which you can't, offline). The metric math itself gets direct unit tests: ROUGE-L is order-aware (reversed word order scores lower than exact, but above zero), F1 handles empty-vs-empty as 1.0, embedding similarity ranks identical > related > unrelated, and the judge parser clamps "10" down to 1.0 and "no score" down to 0.0.

Two expectations in that first draft were flat-out wrong, which is the test suite working: cosine similarity of a vector with itself is 1.0000000000000002 (not exactly 1.0), so the assertion became "close to 1.0 within 1e-6"; and my mental ROUGE-L on reversed word order was miscalculated — the LCS of "a b c d" and "d c b a" is 1, an F of 0.25, not the >0.4 I'd guessed. The tests caught both before the PR shipped.

Design Notes

• Offline by default. Every metric except the judge is deterministic string/vector math; tests swap in fake chat and embedding models, so the whole suite (now 73 tests) runs with no network and no key.

• Small golden sets. Three questions per dataset is enough to catch regressions and keep runs sub-second. Growing to dozens makes the suite flaky and slow.

• Metrics as interchangeable scores. A List<Metric> is passed to evaluate, so a future report could weight judge differently or drop exact entirely without touching the service.

• Defensive parsing. The judge prompt says "reply with only a single integer" because models don't; the regex+clamp makes the metric robust to anything it actually returns.

Next Steps

Evaluation gives us a yardstick, which is the precondition for the remaining checklist items: advanced features (function calling, multi-modal, structured output) and LLM integration (swapping in Anthropic/Google/Ollama and comparing models). With /eval in place, those comparisons will have numbers instead of vibes.

Resources

• LangChain4j documentation

• Evaluation and Testing Issue

Conclusion

A golden-dataset harness turns "the demo seems fine" into six numbers per capability. The most instructive part of building it was deciding what not to trust: an LLM judge needs regex parsing, string metrics need normalization, and your own intuitions about ROUGE-L need tests. With /eval reporting exact, contains, F1, ROUGE-L, embedding similarity, and judge scores, the next model swap will be measured, not guessed.

The full implementation lives in our GitHub repository.

The Dependencies

Three additions to pom.xml:

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-devtools</artifactId>
    <scope>runtime</scope>
    <optional>true</optional>
</dependency>
<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-actuator</artifactId>
</dependency>
<dependency>
    <groupId>org.springdoc</groupId>
    <artifactId>springdoc-openapi-starter-webmvc-ui</artifactId>
    <version>2.9.0</version>
</dependency>

DevTools is scoped runtime and optional so it never ships in a production artifact. Springdoc 2.9.0 is the latest 2.x line — 3.x targets Spring Boot 4, and we're on 3.5.

DevTools: Auto-Restart

DevTools watches the classpath and restarts the app whenever a source file changes — the modern descendant of spring-boot-devtools' classic auto-restart. While editing the demo, saving a Java file triggers a restart automatically, so the old manual stop/spring-boot:run loop disappears. Nothing else to configure.

Actuator: Monitoring

Actuator exposes health and metrics without any code. The demo opts into a focused set — health, info, and metrics — rather than the kitchen sink:

management.endpoints.web.exposure.include=health,info,metrics
management.endpoint.health.show-details=always
management.info.env.enabled=true
info.app.name=langchain4j-demo
info.app.description=LangChain4j feature demo: memory, RAG, agents, prompting, streaming, WebSocket, DB

/actuator/health now reports the app plus its H2 database component:

{"status":"UP","components":{"db":{"status":"UP","details":{"database":"H2"}},"diskSpace":{"status":"UP"},"ping":{"status":"UP"},"ssl":{"status":"UP"}}}

One configuration gotcha surfaced during this work: management.info.env.enabled is required to expose info.* properties — without it, /actuator/info returns {} no matter how many info.app.* keys you define. That's the kind of thing a one-line test (see below) would have caught on the first run.

Swagger UI: Testing from the Browser

springdoc scans the @RestControllers and generates an OpenAPI document automatically — every endpoint, request record, and response record shows up with schemas. Open the UI at /swagger-ui.html and each of the thirteen /api/** paths becomes a clickable "Try it out" form:

GET  /api/chat/stream  (SSE)
POST /api/chat  {"message": "..."}
POST /api/ask   {"question": "..."}
...and the prompt, search, index, store, and history endpoints

/v3/api-docs serves the raw JSON if you want to feed it to other tools.

Making the docs informative

Auto-generated docs are accurate, but they start out terse — just parameter names and types. So each endpoint got explicit OpenAPI annotations: a @Tag per controller, an @Operation summary and description per method, @Parameter descriptions and examples for query/path params, @Schema descriptions on the request/response records, and @ApiResponse status codes. A tiny OpenApiConfig bean sets the title and description in the UI header.

The result reads like a hand-written API reference that can't drift:

@PostMapping("/chat")
@Operation(summary = "Chat with the memory-backed assistant",
        description = "Sends the message to the assistant with conversation memory. "
                + "Use a conversationId to keep a multi-turn conversation; a new id starts fresh.")
@ApiResponse(responseCode = "200", description = "The assistant's answer",
        content = @Content(schema = @Schema(implementation = ChatResponse.class)))
public ChatResponse chat(@RequestBody ChatRequest request) { ... }

with the request schema annotated on the record:

public record ChatRequest(
        @Schema(description = "Conversation (memory) id; a fresh id starts a new conversation",
                example = "web", defaultValue = "api")
        String conversationId,
        @Schema(description = "The user message or task to run", example = "Hello!")
        String message) { ... }

Every query parameter also carries a description and example, so "Try it out" comes pre-filled with sensible values.

Testing the Tooling

DevToolingTest treats the tooling itself as a feature to verify:

@SpringBootTest(properties = "app.cli.enabled=false")
@AutoConfigureMockMvc
class DevToolingTest {

    @Test
    void actuatorHealthReportsUp() throws Exception {
        mockMvc.perform(get("/actuator/health"))
                .andExpect(status().isOk())
                .andExpect(jsonPath("$.status").value("UP"));
    }

    @Test
    void openApiDocsDescribeTheRestApi() throws Exception {
        mockMvc.perform(get("/v3/api-docs"))
                .andExpect(content().string(containsString("/api/chat")))
                .andExpect(content().string(containsString("/api/chat/stream")));
    }
}

The OpenAPI test is genuinely useful: if someone renames a route, the docs test catches it. All 60 tests pass offline.

Design Notes

• Expose the minimum. health, info, metrics cover monitoring without opening env or beans to the world.

• Let the docs come from the code. Every endpoint we documented by hand in the README previously had drifted (/api/prompt/sentiment was actually /api/sentiment). springdoc derives docs from the running code, so they can't drift — and it flagged the README table to fix.

• Tooling gets tests too. A 4-test class proves health, info, docs, and the UI redirect without a browser.

Next Steps

Back to the feature checklist: evaluation (automated scoring of the typed outputs) and LLM integration (alternative providers and models). Both become more tractable now that every endpoint is testable from Swagger UI and observable through Actuator.

Resources

• Spring Boot Actuator Documentation

• springdoc-openapi

• Developer Tooling Issue

Conclusion

Tooling work is the least glamorous milestone in the demo, but it pays off immediately: DevTools removed the restart loop, Actuator made the app observable with four config lines, and Swagger UI turned "let me check the API" into a browser tab instead of a curl session. The best return, though, was the discovery that hand-maintained docs had already drifted — a good argument for generating documentation from the code that implements it.

The full implementation lives in our GitHub repository.

The Shape of It

Spring Boot 3 already has the web layer; LangChain4j has the AI layer. The integration milestone is mostly about wiring the two together. The application now ships:

• a REST API under /api (chat, RAG, agent, prompt helpers, search, history)

• a Server-Sent Events endpoint for token-by-token streaming

• a WebSocket endpoint for the same, frame-by-frame

• an H2 database, via Spring Data JPA, for conversation history

The pom.xml gains four starters:

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-websocket</artifactId>
</dependency>
<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-data-jpa</artifactId>
</dependency>
<dependency>
    <groupId>com.h2database</groupId>
    <artifactId>h2</artifactId>
    <scope>runtime</scope>
</dependency>

REST API

Controllers are thin: take a request record, call the service that already exists from the earlier milestones, return a response record. The chat endpoint reuses the memory-backed Assistant from the memory milestone, with conversationId defaulting to "api" for curl users:

@PostMapping("/chat")
public ChatResponse chat(@RequestBody ChatRequest request) {
    String answer = assistant.chat(request.conversationId(), request.message());
    return new ChatResponse(answer);
}

Because RAG (/api/ask) and the agent (/api/agent) were built as services in their own milestones, exposing them is a three-liner each. The prompt helpers from the last milestone (/api/prompt/sentiment, /api/prompt/movie, /api/prompt/topics) and semantic search (/api/search) get the same treatment. A static index.html lists every endpoint, and chat.html is a working browser client.

Streaming with Server-Sent Events

Streaming needed something we didn't have yet: a StreamingChatModel. That's a second bean in AiConfig, alongside the existing chat model:

@Bean
public StreamingChatModel streamingChatModel() {
    return OpenAiStreamingChatModel.builder()
            .apiKey(System.getenv("OPENAI_API_KEY"))
            .modelName("gpt-4o-mini")
            .build();
}

The streaming logic itself lives in a small service with no web-layer dependencies. That separation is deliberate — it means the same code serves both the SSE endpoint and the WebSocket handler, and it means we can unit-test it with a fake model:

@Service
public class ChatStreamingService {

    public void stream(String message, StreamConsumer consumer) {
        streamingChatModel.chat(ChatRequest.builder()
                .messages(List.of(
                        SystemMessage.from("You are a helpful assistant. Answer concisely."),
                        UserMessage.from(message)))
                .build(), new StreamingChatResponseHandler() {
                    @Override
                    public void onPartialResponse(String partialResponse) {
                        consumer.onToken(partialResponse);
                    }
                    @Override
                    public void onCompleteResponse(ChatResponse completeResponse) {
                        consumer.onComplete(completeResponse.aiMessage().text());
                    }
                    @Override
                    public void onError(Throwable error) {
                        consumer.onError(error);
                    }
                });
    }
}

The consumer is a plain interface: onToken, onComplete, onError. The SSE endpoint adapts it to an SseEmitter:

@GetMapping(value = "/chat/stream", produces = MediaType.TEXT_EVENT_STREAM_VALUE)
public SseEmitter stream(@RequestParam String message) {
    SseEmitter emitter = new SseEmitter(60_000L);
    streamingService.stream(message, new ChatStreamingService.StreamConsumer() {
        @Override public void onToken(String token) {
            try {
                emitter.send(SseEmitter.event().data(token));
            } catch (IOException e) {
                emitter.completeWithError(e);
            }
        }
        @Override public void onComplete(String fullText) { emitter.complete(); }
        @Override public void onError(Throwable error) { emitter.completeWithError(error); }
    });
    return emitter;
}

The browser client reads the stream with a plain EventSource — no library needed.

WebSocket

The WebSocket handler is the same streaming service wearing a different transport. The client sends a JSON message; the server replies with one text frame per token and a final [DONE] frame:

@Component
public class ChatWebSocketHandler extends TextWebSocketHandler {

    @Override
    protected void handleTextMessage(WebSocketSession session, TextMessage message) {
        ChatMessagePayload payload;
        try {
            payload = objectMapper.readValue(message.getPayload(), ChatMessagePayload.class);
        } catch (IOException e) {
            sendQuietly(session, new TextMessage("[ERROR] Invalid message payload: " + e.getMessage()));
            return;
        }
        streamingService.stream(payload.message(), new ChatStreamingService.StreamConsumer() {
            @Override public void onToken(String token) {
                sendQuietly(session, new TextMessage(token));
            }
            @Override public void onComplete(String fullText) {
                sendQuietly(session, new TextMessage("[DONE]"));
            }
            @Override public void onError(Throwable error) {
                sendQuietly(session, new TextMessage("[ERROR] " + error.getMessage()));
            }
        });
    }
}

WebSocketConfig registers the handler at /ws/chat and opens CORS for a browser client:

@Configuration
@EnableWebSocket
public class WebSocketConfig implements WebSocketConfigurer {

    @Override
    public void registerWebSocketHandlers(WebSocketHandlerRegistry registry) {
        registry.addHandler(chatWebSocketHandler, "/ws/chat").setAllowedOrigins("*");
    }
}

Database Integration

The last piece: conversation history moved out of memory and into a real store. The entity is a plain JPA @Entity; the repository is one interface with derived queries:

public interface ConversationEntryRepository extends JpaRepository<ConversationEntry, Long> {
    List<ConversationEntry> findByConversationIdOrderByTimestampAsc(String conversationId);
    List<String> findDistinctConversationIds();
    void deleteByConversationId(String conversationId);
}

The service wraps it with @Transactional — the derived deleteByConversationId needs a real transaction, something the test suite caught on the first run:

@Service
@Transactional
public class ConversationHistoryService {
    public ConversationEntry record(String conversationId, String role, String text) { ... }
    public List<ConversationEntry> history(String conversationId) { ... }
    public List<String> conversationIds() { ... }
    public void clear(String conversationId) { ... }
}

The REST layer exposes it as GET /api/history, GET /api/history/{id}, and DELETE /api/history/{id}. The database is in-memory H2 (jdbc:h2:mem:demo), so it resets on restart — but it's a real database with real JPA semantics, and swapping in Postgres would just be a JDBC URL and driver change.

Keeping It Testable Offline

The web layer added a second AI surface — streaming — which needed a fake just like FakeChatModel. FakeStreamingChatModel implements StreamingChatModel and plays back a fixed list of tokens:

class FakeStreamingChatModel implements StreamingChatModel {

    private final List<String> tokens;

    FakeStreamingChatModel(String... tokens) {
        this.tokens = List.of(tokens);
    }

    @Override
    public void doChat(ChatRequest request, StreamingChatResponseHandler handler) {
        StringBuilder full = new StringBuilder();
        for (String token : tokens) {
            handler.onPartialResponse(token);
            full.append(token);
        }
        handler.onCompleteResponse(ChatResponse.builder()
                .aiMessage(AiMessage.from(full.toString()))
                .build());
    }
}

With that in place the tests split cleanly by layer:

• ChatStreamingServiceTest — token forwarding and completion, unit-level

• ChatWebSocketHandlerTest — frames arrive in order followed by [DONE], and a bad payload yields [ERROR]

• ConversationHistoryServiceTest — @DataJpaTest: ordering, empty history, distinct conversation ids, scoped deletes

• ChatApiControllerTest — @SpringBootTest + @AutoConfigureMockMvc with @MockitoBean stand-ins for the services; the SSE test uses asyncDispatch on the returned MvcResult

• HistoryApiControllerTest — end-to-end through the real H2-backed service

All 56 tests run offline.

Design Notes

• Streaming is transport-agnostic. ChatStreamingService knows nothing about SseEmitter or WebSocketSession; both transports are thin adapters over the same StreamConsumer. That is what makes it unit-testable and gives SSE and WebSocket identical behavior for free.

• Reuse beats new abstractions. The controller didn't reinvent RAG or agents — it calls the existing QaService and ChainService.

• Transactions are not optional. Derived delete queries in Spring Data JPA require a transaction; the failing test on the first run was the cheapest possible proof.

• Fakes scale with the surface. One fake per AI boundary (FakeChatModel, FakeEmbeddingModel, now FakeStreamingChatModel) keeps every layer of the app testable without API keys.

Next Steps

From our checklist: evaluation (now that prompts and outputs are typed, they're ready to be scored automatically) and LLM integration (swapping in other providers and models). Also worth exploring: multi-modal input and the new agentic module for advanced orchestration.

Resources

• Official LangChain4j Streaming Documentation

• Spring Boot SSE and WebSocket

• Integration Features Issue

Conclusion

Integration was the least "AI" milestone in the whole demo — no new prompting tricks, no model gymnastics. But it's the one that turns a library into an application: a browser can now chat, stream, and ask questions against every feature built so far, and conversation state lives in a real database. The design payoff was architectural: keeping streaming logic transport-free meant SSE and WebSocket both shipped as thin adapters, and keeping every fake at the model boundary meant all of it stays offline-testable.

The full implementation lives in our GitHub repository.

Prompt Templates

A prompt template is a prompt skeleton with {{placeholders}}. Instead of string-concatenating prompts in handlers all over the codebase, you define the template once and render it with variables.

LangChain4j's PromptTemplate is a pure, offline component — no model involved. It renders a template into a Prompt, which converts to a system or user message:

@Service
public class PromptService {

    private static final String SYSTEM_TEMPLATE = """
            You are a professional movie critic.
            Always write your reviews in a {{tone}} tone.
            """;

    private static final String USER_TEMPLATE = """
            Write a short review for the movie "{{movie}}" ({{year}}).
            Include a rating out of 10 in your review.
            """;

    public List<ChatMessage> buildMovieReviewMessages(String movie, int year, String tone) {
        Prompt systemPrompt = PromptTemplate.from(SYSTEM_TEMPLATE).apply(Map.of("tone", tone));
        Prompt userPrompt = PromptTemplate.from(USER_TEMPLATE).apply(Map.of("movie", movie, "year", year));
        return List.of(systemPrompt.toSystemMessage(), userPrompt.toUserMessage());
    }
}

The CLI command /template prints the fully rendered messages without calling any API:

/template Inception
Rendered prompt template for "Inception" (year 2010, enthusiastic tone):

SYSTEM:
You are a professional movie critic.
Always write your reviews in an enthusiastic tone.

USER:
Write a short review for the movie "Inception" (2010).
Include a rating out of 10 in your review.

Because rendering is deterministic, it is fully unit-testable: same template + same variables → same messages. We assert exactly that in PromptServiceTest.

AI Services take templates one step further: @SystemMessage and @UserMessage accept templates whose variables are resolved from method parameters (via @V, or plain parameter names with Spring's -parameters):

@UserMessage("Write a review of {{movie}} from {{year}}")
String review(@V("movie") String movie, @V("year") int year);

Few-Shot Learning Examples

Sometimes a short instruction isn't enough — the model benefits from seeing correct input/output pairs. Few-shot prompting embeds a handful of labeled examples into the prompt. Our FewShotAssistant classifies sentiment into POSITIVE / NEGATIVE / NEUTRAL using a mini dataset in the system message:

public interface FewShotAssistant {

    @SystemMessage("""
            You are a sentiment classifier. Classify the sentiment of the given text
            as one of: POSITIVE, NEGATIVE, NEUTRAL. Reply with exactly one of these words.

            Examples:
            Text: "I absolutely loved this movie, best film of the year!"
            Sentiment: POSITIVE

            Text: "This restaurant is terrible, the food was cold and the service rude."
            Sentiment: NEGATIVE

            Text: "The package arrived on time. Nothing more to say."
            Sentiment: NEUTRAL
            """)
    Sentiment classify(@UserMessage String text);
}

Three well-chosen examples do double duty: they pin down the format (exactly one word) and illustrate the decision boundary (contrasting strong opinions with a neutral, factual statement). AiServices renders the system message with the examples on every call.

Output Parsers (Structured Output)

The most robust prompting technique is making the model return a typed value instead of free text. In LangChain4j this is done by choosing the AI Service's return type — the framework derives a schema, requests JSON matching it, and parses the reply into an object. The internal machinery is OutputParser (EnumOutputParser, PojoOutputParser, StringListOutputParser, and friends); we never touch it directly, we just pick the return type.

Returning an enum — the sentiment above returns Sentiment, parsed into the matching constant.

Returning a POJO — MovieExtractor returns a MovieReview record:

public record MovieReview(String title, int year, String director, double rating, String summary) {}

public interface MovieExtractor {
    @SystemMessage("""
            Extract information about the movie from the given text.
            Return the data as a JSON object with exactly these fields:
            title (string), year (integer), director (string), rating (number from 1 to 10), summary (string).
            """)
    MovieReview extract(@UserMessage String text);
}

The handler gets an object, not a string to hand-parse:

/movie Inception is a 2010 film directed by Christopher Nolan about planting ideas in dreams.
Movie > MovieReview[title=Inception, year=2010, director=Christopher Nolan, rating=9.0, summary=...]

Returning a collection — TopicExtractor returns List<String>. (Collection-of-strings output is parsed as one item per line, so the system message tells the model that format.)

Testing Without a Live Model

Prompt engineering is all about iteration, so offline testing is a big win. We added a second shared test helper, FakeChatModel, which mirrors FakeEmbeddingModel: it implements the ChatModel interface, captures the exact ChatRequest LangChain4j builds, and returns a canned reply. That lets tests verify two things at once:

1. What goes in — the generated system/user messages contain the few-shot examples and the interpolated template variables.

2. What comes out — the canned reply is parsed into the correct enum constant, record, or list.

@Test
void embedsFewShotExamplesInTheSystemMessage() {
    FakeChatModel chatModel = new FakeChatModel("POSITIVE");
    FewShotAssistant assistant = AiServices.builder(FewShotAssistant.class)
            .chatModel(chatModel)
            .build();

    Sentiment sentiment = assistant.classify("This movie is amazing!");

    assertThat(chatModel.lastSystemMessage())
            .contains("Examples:")
            .contains("Sentiment: NEGATIVE");
    assertThat(sentiment).isEqualTo(Sentiment.POSITIVE);
}

@Test
void parsesTheModelReplyIntoTheMovieReviewRecord() {
    FakeChatModel chatModel = new FakeChatModel("""
            {"title": "Inception", "year": 2010, "director": "Christopher Nolan",
             "rating": 9.0, "summary": "A thief enters dreams to plant an idea."}
            """);
    MovieReview review = AiServices.builder(MovieExtractor.class)
            .chatModel(chatModel)
            .build()
            .extract("Tell me about Inception.");

    assertThat(review.title()).isEqualTo("Inception");
    assertThat(review.rating()).isEqualTo(9.0);
}

These tests exercise the real AiServices proxy generation and the real output parsers — the only substitute is the model itself. Combined with PromptServiceTest, the whole prompting layer is verified without a single network call.

Design Notes

• Templates are the API, not an afterthought. PromptTemplate and @UserMessage/@SystemMessage templates keep prompts next to their consumers and make variables explicit.

• Examples live in the system message. They ship with every call and double as both format and boundary guidance.

• Output contracts beat prompt begging. Telling the model "return JSON" is unreliable; returning MovieReview makes the schema explicit and the parsing automatic and typed.

• FakeChatModel closes the loop. It lets us assert on both the rendered prompt and the parsed result, which is where most prompt-engineering bugs actually live.

Next Steps

From our checklist: LLM integration (pluggable providers/models), integration features (REST endpoints, streaming), and evaluation — now that outputs are typed (Sentiment, MovieReview, List<String>), they become much easier to evaluate automatically.

Resources

• Official LangChain4j Prompting Documentation

• LangChain4j Structured Output

• Prompting Techniques Issue

Conclusion

Prompting is the part of an LLM application you will iterate on most, so it pays to make it structured. Templates keep prompts DRY and inspectable; few-shot examples teach format and boundaries with zero extra code; typed return values turn model output from a string to parse into a contract to rely on. And because all three are pure or deterministic, they test cleanly offline — which is exactly what you want when the thing you're tuning changes every day.

The full implementation lives in our GitHub repository.

Two Kinds of "Chains"

The word "chain" means two different things in the LLM world, and LangChain4j maps them to two different mechanisms:

1. Processing chains — we decide the steps, in code. Input flows through fixed stages: preprocess, route, transform, postprocess. Deterministic and testable.

2. Agentic loops — the model decides the steps, calling functions in a loop ("reason, call tool, observe result") until the task is done. Non-deterministic; correctness is delegated to the model.

LangChain4j implements the second one as function calling under the hood (OpenAI-style tool calls), and wraps it in AiServices. We implement the first one ourselves as plain Java composition.

Tools: The Agent's Hands

A tool is an ordinary method annotated with @Tool. The chat model sees a description of each tool and its parameters, and decides when to call it. AiServices executes the call and feeds the result back to the model — automatically, in a loop, with no glue code from us.

Our agent gets three tools. First, an arithmetic calculator. The trickiest part is safety: we never want to evaluate arbitrary model-produced strings as code, so we wrote a small recursive-descent parser that only accepts digits, + - * /, parentheses, and dots:

@Component
public class CalculatorTool {

    @Tool("Calculates the result of an arithmetic expression using +, -, *, / and parentheses")
    public double calculate(@P("The arithmetic expression to evaluate, e.g. \"(1 + 2) * 3\"") String expression) {
        if (expression == null || expression.isBlank()) {
            throw new IllegalArgumentException("Expression must not be empty");
        }
        return new Evaluator(expression).evaluate();
    }
}

@Tool marks the callable function; @P documents a single parameter (LangChain4j turns these into the model's function schema). Second, a document-search tool that gives the agent access to the same embedded knowledge base the RAG pipeline uses:

@Tool("Searches the indexed documents and returns the most relevant passages with their relevance scores")
public String searchDocuments(@P("The search query, e.g. \"what does the document say about RAG\"") String query) {
    List<EmbeddingMatch<TextSegment>> matches = searchService.search(query);
    if (matches.isEmpty()) {
        return "No matching documents found. Documents may need to be indexed first with /index.";
    }
    // ... format matches with scores
}

And third, a stats tool that reports the embedding store's model and size.

The Agent: AiServices with Tools

An agent is just an AI Service interface whose proxy gets built with .tools(...):

public interface Agent {

    @SystemMessage("""
            You are an agent that accomplishes the user's task using the available tools.
            Use the "searchDocuments" tool when the task asks about the indexed documents or your own data.
            Use the "calculate" tool for arithmetic computations.
            If the task does not need a tool, answer directly. Be concise.
            """)
    String execute(@MemoryId String memoryId, @UserMessage String task);
}

Wired in AiConfig alongside the other AI services, reusing the same per-conversation memory provider:

@Bean
public Agent agent(ChatModel chatModel,
                   CalculatorTool calculatorTool,
                   DocumentSearchTool documentSearchTool,
                   EmbeddingStoreStatsTool storeStatsTool,
                   ChatMemoryRegistry chatMemoryRegistry,
                   ...) {
    return AiServices.builder(Agent.class)
            .chatModel(chatModel)
            .chatMemoryProvider(createChatMemoryProvider(chatMemoryRegistry, modelName, maxMessages, maxTokens))
            .tools(calculatorTool, documentSearchTool, storeStatsTool)
            .build();
}

The system message is important: it tells the model which tool fits which kind of task. Without it the model still sees the tool schemas, but good guidance reduces wasted calls.

The Chain: Deterministic Routing

ChainService is our processing chain — a pipeline whose stages we control:

@Service
public class ChainService {

    private final Agent agent;
    private final CalculatorTool calculatorTool;

    public String ask(String memoryId, String task) {
        String normalized = task.trim();
        if (CalculatorTool.isArithmetic(normalized)) {
            return "Result: " + calculatorTool.calculate(normalized);
        }
        return agent.execute(memoryId, normalized);
    }
}

Stage 1 (preprocess + route): if the task is a pure numeric expression, resolve it locally — no model call, no tokens burned, and no chance of the LLM hallucinating a sum. Stage 2 (execute): everything else — including worded questions like "what is two plus two?" — goes to the agent, which can still decide to call the calculator itself.

This hybrid is a nice demonstration of why chains and agents are complements, not competitors: cheap, deterministic stages handle the obvious cases; the flexible, model-driven loop handles the rest.

Trying It Out

/agent compute (20 * 5) - 8
Agent > Result: 92.0

Pure arithmetic never touches the model — the chain resolves it. Now a worded version that needs the agent and its document-search tool (after /index sample-data):

/agent what do the documents say about agents?
Agent > The documents describe agents as programs that use tools to complete tasks:
        "Agents use tools to complete tasks to achieve a goal." (from langchain4j.txt)

Behind the scenes the model called searchDocuments, saw the retrieved passages, and answered from them — the RAG loop, but driven by the model's own decision instead of a fixed augmentor.

Testing Without a Live Model

The calculator and the chain are deterministic, so they get real unit tests — no mocks, no network:

@Test
void respectsOperatorPrecedence() {
    assertThat(calculatorTool.calculate("2 + 3 * 4")).isEqualTo(14.0);
}

@Test
void routesArithmeticToTheCalculatorWithoutTheAgent() {
    ChainService chain = new ChainService((memoryId, task) -> "agent called: " + task, calculatorTool);
    assertThat(chain.ask("main", " 2 + 3 * 4 ")).isEqualTo("Result: 14.0");
}

@Test
void delegatesWordedTasksToTheAgent() {
    ChainService chain = new ChainService((memoryId, task) -> "agent handled \"" + task + "\"", calculatorTool);
    assertThat(chain.ask("main", "What can you do?")).isEqualTo("agent handled \"What can you do?\"");
}

Because Agent is an interface, the chain test substitutes a lambda — asserting both the routing and that the memory id is passed through, entirely offline. The document-search tool is tested with the same FakeEmbeddingModel we use elsewhere: index real text, call the tool, assert it returns ranked passages.

Design Notes

• @Tool on methods, @P on parameters. That's the whole declarative surface for function calling in LangChain4j. Parameters map to the OpenAI-style schema the model uses to format its calls.

• Tools are Spring beans. The three tool classes are @Components with their own dependencies (the search tool reuses SemanticSearchService), so they compose cleanly with the rest of the context.

• Memory composes for free. The agent uses the same ChatMemoryProvider as the assistant and QA services, so it can hold a multi-turn, tool-using conversation with @MemoryId scoping.

• Safety matters. Never evaluate strings the model returns as code. Our evaluator parses a grammar, rejects unknown characters, throws on malformed input and division by zero, and is covered by tests for exactly those failure modes.

Next Steps

The natural follow-ups from our checklist: prompting techniques (few-shot examples, templates) and evaluation (benchmarking whether the agent's tool choices are good ones). There is also a new, explicitly experimental langchain4j-agentic module in LangChain4j with structured "agent definitions" — worth a look once the stable AiServices + @Tool path has proven itself here.

Resources

• Official LangChain4j AI Services Documentation

• LangChain4j Tools / Function Calling

• Chains and Agents Issue

Conclusion

Chains and agents are two complementary ways to make an LLM application do things instead of just say things. Deterministic processing chains give us fast, testable, token-cheap routing for the cases we can predict; AiServices with @Tool gives the model the ability to call our own code when it decides that's needed — whether that's arithmetic, searching the knowledge base, or anything else we want to expose as a function. Both compose with the memory and RAG infrastructure we already built, which is exactly the point of a demo that grows feature by feature.

The full implementation lives in our GitHub repository.

The Document Pipeline

LangChain4j models the journey from raw file to searchable chunk as a chain of small steps:

1. Load — read bytes from a file or URL (FileSystemDocumentLoader).

2. Parse — turn bytes into text using a DocumentParser chosen for the format (PDFBox for PDFs, plain text otherwise).

3. Split — break the text into TextSegments with a DocumentSplitter.

4. Transform — enrich each segment (we prepend the source file name).

Every step is swappable. We used to do this inside SemanticSearchService with hardcoded settings; now it lives in a dedicated DocumentService so the processing concerns are separate from the embedding concerns.

Parsing PDFs

PDF parsing needs a parser. LangChain4j has several; we added the Apache PDFBox one:

<dependency>
    <groupId>dev.langchain4j</groupId>
    <artifactId>langchain4j-document-parser-apache-pdfbox</artifactId>
</dependency>

(The BOM manages its version, so no version tag is needed.)

ApachePdfBoxDocumentParser extracts text from a PDF using PDFBox. DocumentService picks the right parser per extension — PDFs get PDFBox, everything else gets the plain-text parser:

private DocumentParser parserFor(Path path) {
    String name = path.getFileName().toString().toLowerCase(Locale.ROOT);
    if (name.endsWith(".pdf")) {
        return new ApachePdfBoxDocumentParser();
    }
    return new TextDocumentParser();
}

And loads a single file or a whole directory:

public List<TextSegment> loadAndSplit(Path filePath) {
    return split(FileSystemDocumentLoader.loadDocument(filePath, parserFor(filePath)));
}

public List<TextSegment> loadAndSplitDirectory(Path directoryPath) {
    try (Stream<Path> files = Files.list(directoryPath)) {
        return files.filter(Files::isRegularFile)
                .filter(file -> !file.getFileName().toString().startsWith("."))
                .flatMap(file -> loadAndSplit(file).stream())
                .toList();
    } catch (IOException e) {
        throw new RuntimeException("Failed to list documents in " + directoryPath, e);
    }
}

Note that we skip hidden files (like macOS .DS_Store) — a small but necessary bit of processing hygiene. The loader attaches useful Metadata automatically, including file_name and the absolute directory path.

Splitting Strategies

LangChain4j ships several DocumentSplitters out of the box:

• DocumentByParagraphSplitter — chunks by paragraphs (blank-line separated blocks)

• DocumentByLineSplitter — chunks by lines

• DocumentBySentenceSplitter — chunks by sentences (OpenNLP sentence detection)

• DocumentByWordSplitter — chunks by words

• DocumentByCharacterSplitter — fixed-size character chunks

• DocumentSplitters.recursive(...) — paragraphs first, falling back to lines, then sentences, then words for anything still too big

We wrapped these in a small enum so the strategy is configurable and switchable at runtime:

public enum DocumentSplitterType {
    RECURSIVE("recursive"),
    PARAGRAPH("paragraph"),
    LINE("line"),
    SENTENCE("sentence"),
    WORD("word"),
    CHARACTER("character");

    public DocumentSplitter create(int maxChunkSize, int maxOverlap) {
        return switch (this) {
            case RECURSIVE -> DocumentSplitters.recursive(maxChunkSize, maxOverlap);
            case PARAGRAPH -> new DocumentByParagraphSplitter(maxChunkSize, maxOverlap);
            case LINE -> new DocumentByLineSplitter(maxChunkSize, maxOverlap);
            case SENTENCE -> new DocumentBySentenceSplitter(maxChunkSize, maxOverlap);
            case WORD -> new DocumentByWordSplitter(maxChunkSize, maxOverlap);
            case CHARACTER -> new DocumentByCharacterSplitter(maxChunkSize, maxOverlap);
        };
    }
}

DocumentService uses the selected strategy together with a configurable chunk size and overlap:

private List<TextSegment> split(Document document) {
    String fileName = document.metadata().getString(FILE_NAME);
    return splitterType.create(maxChunkSize, maxOverlap).split(document).stream()
            .map(segment -> withFileNamePrefix(segment, fileName))
            .toList();
}

Why the file-name prefix?

Each splitter copies the document's metadata into every TextSegment. On top of that we prepend the file name to the segment text. This is a documented LangChain4j trick: retrieval improves when every chunk knows where it came from, because the embedding of e.g. rag.pdf\nRetrieval Augmented Generation combines... is closer to a query about RAG than the bare sentence would be.

Wiring It In

SemanticSearchService now delegates the load-parse-split stage and keeps only the embed-and-store stage:

public int indexDocument(Path filePath) {
    return indexSegments(documentService.loadAndSplit(filePath));
}

public int indexDirectory(Path directoryPath) {
    return indexSegments(documentService.loadAndSplitDirectory(directoryPath));
}

private int indexSegments(List<TextSegment> segments) {
    if (segments.isEmpty()) {
        return 0;
    }
    int sizeBefore = embeddingStore.size();
    List<Embedding> embeddings = embeddingModel.embedAll(segments).content();
    embeddingStore.addAll(embeddings, segments);
    save();
    return embeddingStore.size() - sizeBefore;
}

A nice side effect: without an in-place splitter anymore, SemanticSearchService just embeds and stores whatever segments it is given — the splitter lives where it belongs, in DocumentService.

Trying It Out

We added a sample PDF (sample-data/rag.pdf) and a /splitter command to the CLI. Index the directory — PDFs and text files alike:

/index sample-data
Indexed 7 segment(s). Store now holds 7 embedding(s).

Then ask a question grounded in the PDF:

/ask how does RAG combine a vector database with a chat model?
RAG > RAG combines a vector database with a chat model so the model can answer
      questions using your own documents instead of relying only on training data.

Switch splitting strategies and see how the chunking changes:

/splitter
Splitter type: recursive
Max chunk size: 200 chars
Max overlap   : 20 chars

/splitter paragraph
Switched splitter to 'paragraph'. Re-index documents to re-chunk them with the new strategy.

Testing Without a Live Model

Processing is fully testable offline. The test suite generates a real PDF at runtime with PDFBox, then asserts the parser extracts the expected text:

@Test
void parsesPdfDocuments() throws Exception {
    Path pdf = tempDir.resolve("guide.pdf");
    createPdf(pdf, "Retrieval augmented generation lets a chat model answer from your own documents.");

    DocumentService service = new DocumentService("recursive", 200, 20);

    List<TextSegment> segments = service.loadAndSplit(pdf);

    assertThat(segments).isNotEmpty();
    assertThat(segments.get(0).text()).contains("Retrieval augmented generation");
}

Another test asserts the file-name prefix lands on every segment, and a third proves smaller chunk sizes yield more segments — all without a single network call.

Configuration

Next Steps

Now that documents are first-class, the RAG question-answering from the previous post gets much better input. Natural next explorations: DocumentBySentenceSplitter tuning, EmbeddingStoreIngestor's textSegmentTransformer for richer metadata, and — later in our checklist — chains and agents that operate on the processed documents.

Resources

• Official LangChain4j RAG Documentation

• Apache PDFBox

• Document Processing Issue

Conclusion

Document processing is where RAG becomes real: PDF parsing, per-format parsers, and configurable splitting turn arbitrary files into clean, retrievable chunks. With LangChain4j's DocumentParser and DocumentSplitter interfaces, each stage is a small pluggable component — and with a dedicated DocumentService, the demo now separates how documents become segments from how segments become searchable embeddings.

The full implementation lives in our GitHub repository.

The RAG Pipeline

RAG adds three stages in front of a normal chat call:

1. Query transformation — optionally rewrite the user's question (e.g. to compress a follow-up into a standalone query).

2. Retrieval — embed the question and pull the most similar chunks out of the vector store.

3. Aggregation + injection — combine the hits and append them to the user message before it reaches the model.

LangChain4j models this as an interface: RetrievalAugmentor. The default implementation, DefaultRetrievalAugmentor, composes pluggable pieces — a QueryTransformer, a ContentRetriever, a ContentAggregator, and a ContentInjector. We only need to supply the retriever; sensible defaults handle the rest.

A ContentRetriever That Uses Our Search Service

The natural place to plug RAG in is our SemanticSearchService from the embeddings post. It owns both the embedding model and the store, so the retriever just delegates to it:

public class SemanticSearchContentRetriever implements ContentRetriever {

    private final SemanticSearchService searchService;
    private final int maxResults;

    public SemanticSearchContentRetriever(SemanticSearchService searchService, int maxResults) {
        this.searchService = searchService;
        this.maxResults = maxResults;
    }

    @Override
    public List<Content> retrieve(Query query) {
        return searchService.search(query.text(), maxResults).stream()
                .map(this::toContent)
                .toList();
    }

    private Content toContent(EmbeddingMatch<TextSegment> match) {
        return Content.from(match.embedded(), Map.of(ContentMetadata.SCORE, match.score()));
    }
}

Delegating keeps a single source of truth: the store and the current embedding model live in the service, so a runtime /model switch is picked up by the RAG flow automatically. LangChain4j also ships a ready-made EmbeddingStoreContentRetriever if you'd rather wire the store directly.

Assembling the Augmentor

In AiConfig we expose the retriever as a bean, wrap it in a RetrievalAugmentor, and attach the augmentor to a dedicated question-answering AI service:

@Bean
public ContentRetriever contentRetriever(SemanticSearchService searchService,
                                         @Value("${app.rag.max-results:5}") int maxResults) {
    return new SemanticSearchContentRetriever(searchService, maxResults);
}

@Bean
public RetrievalAugmentor retrievalAugmentor(ContentRetriever contentRetriever) {
    return DefaultRetrievalAugmentor.builder()
            .contentRetriever(contentRetriever)
            .build();
}

@Bean
public QaAssistant qaAssistant(ChatModel chatModel,
                               RetrievalAugmentor retrievalAugmentor,
                               ...) {
    return AiServices.builder(QaAssistant.class)
            .chatModel(chatModel)
            .chatMemoryProvider(createChatMemoryProvider(...))
            .retrievalAugmentor(retrievalAugmentor)
            .build();
}

The QA assistant is a plain AI service interface, just like the chat assistant — the only difference is the augmentor:

public interface QaAssistant {

    @SystemMessage("""
            You are a question-answering assistant that answers only from the provided context.
            Answer the question using the information supplied in the user message. If the context
            does not contain the answer, respond with "I don't know". Keep the answer concise.
            """)
    String ask(@MemoryId String memoryId, @UserMessage String question);
}

The @MemoryId gives each conversation its own memory, so follow-up questions work across turns.

How the Pieces Fit Together

When ask() is called, AiServices runs the augmentor before hitting the model. The DefaultRetrievalAugmentor:

• passes the question through the default QueryTransformer unchanged,

• hands it to our SemanticSearchContentRetriever, which embeds it and searches the store,

• aggregates the results with the default ContentAggregator,

• and injects them into the user message via the default ContentInjector, producing something like:

How does RAG work?

Answer using the following information:
Retrieval Augmented Generation combines a vector database with a chat model ...

Then the chat model answers using only that context — grounded in your documents, not its own guesses.

Trying It Out

The CLI gained an /ask command that chains retrieval and generation. With the bundled sample documents indexed:

/index sample-data
Indexed 6 segment(s). Store now holds 6 embedding(s).

/ask what is a vector database?
RAG > A vector database stores embeddings (numerical representations of text)
      so that similar meanings can be found quickly by vector similarity.

Ask a follow-up and the memory kicks in:

/ask why does that matter for answering questions?
RAG > It lets the model find relevant information from your own documents
      instead of relying only on what it learned during training.

Because retrieval happens per question, the answers are grounded in the indexed material — and with an empty store the augmentor returns nothing, so the assistant honestly says it doesn't know rather than hallucinating.

Testing the Pipeline Without a Live Model

The best part of separating retrieval from generation is that retrieval is fully testable offline. With the deterministic FakeEmbeddingModel from the embeddings post, we index a document, then assert that the augmentor retrieves the right chunk and injects it into the user message:

SemanticSearchService searchService = new SemanticSearchService(
        modelName -> new FakeEmbeddingModel(), "test", null, 5);
searchService.indexDirectory(docs);

ContentRetriever retriever = new SemanticSearchContentRetriever(searchService, 5);
RetrievalAugmentor augmentor = DefaultRetrievalAugmentor.builder()
        .contentRetriever(retriever)
        .build();

AugmentationResult result = augmentor.augment(new AugmentationRequest(
        UserMessage.from("How does RAG work?"),
        Metadata.from(question, "qa", List.of())));

assertThat(result.contents()).isNotEmpty();
assertThat(((UserMessage) result.chatMessage()).singleText())
        .contains("Retrieval Augmented Generation");

This exercises the whole RAG flow — transform, retrieve, aggregate, inject — with zero network calls, which is exactly the kind of test you want before paying for model invocations.

Configuration

Next Steps

Our RAG is grounded, but it's the simplest flavor. LangChain4j has more pieces to explore: CompressingQueryTransformer (rewrites follow-up questions using chat memory), re-ranking aggregators, and dedicated vector databases like pgvector or Elasticsearch that replace the in-memory store at scale. Document processing (parsing PDFs, splitting smarter) is the other natural next step.

Resources

• Official LangChain4j RAG Documentation

• GitHub Examples Repository

• RAG Issue

Conclusion

RAG turns a general-purpose chatbot into one that answers from your own knowledge base. With LangChain4j, that's a ContentRetriever, a RetrievalAugmentor, and one line on the AiServices builder — and because the pipeline is composed of small interfaces, the retrieval half is testable without any API calls. Combined with conversation memory and semantic search from earlier posts, the demo is now a complete grounded question-answering system.

The full implementation lives in our GitHub repository.

What Is an Embedding?

An embedding is a vector (a list of numbers) that represents the meaning of a piece of text. Texts with similar meanings end up with similar vectors — they sit close together in a high-dimensional space. "How do I fix my computer" and "my laptop is broken" will have nearby vectors, even though they share almost no words.

Semantic search works in two phases:

1. Indexing — split documents into chunks, embed each chunk, and store the vectors in a vector store.

2. Querying — embed the user's question and find stored vectors that are closest to it (using cosine similarity).

LangChain4j gives us the building blocks: EmbeddingModel (embeds text), EmbeddingStore (stores vectors), EmbeddingStoreIngestor (runs the indexing pipeline), and DocumentSplitters (chunks documents).

The Semantic Search Service

We added a SemanticSearchService that ties these pieces together. It's a Spring component that holds an EmbeddingModel and an InMemoryEmbeddingStore, which we persist to a JSON file.

@Service
public class SemanticSearchService {

    @Autowired
    public SemanticSearchService(Function<String, EmbeddingModel> modelFactory,
                                 @Value("${app.embedding.model-name:text-embedding-3-small}") String modelName,
                                 @Value("${app.embedding.store-path:}") String storePath,
                                 @Value("${app.embedding.max-results:5}") int defaultMaxResults) {
        this.modelFactory = modelFactory;
        this.modelName = modelName;
        this.embeddingModel = modelFactory.apply(modelName);
        this.storePath = storePath == null || storePath.isBlank() ? null : Path.of(storePath);
        this.defaultMaxResults = defaultMaxResults;
        this.embeddingStore = loadStore();
    }

    public int indexDirectory(Path directoryPath) {
        return ingest(FileSystemDocumentLoader.loadDocuments(directoryPath));
    }

    private int ingest(List<Document> documents) {
        int sizeBefore = embeddingStore.size();
        EmbeddingStoreIngestor.builder()
                .documentSplitter(DocumentSplitters.recursive(200, 20))
                .embeddingModel(embeddingModel)
                .embeddingStore(embeddingStore)
                .build()
                .ingest(documents);
        save();
        return embeddingStore.size() - sizeBefore;
    }

    public List<EmbeddingMatch<TextSegment>> search(String query) {
        Response<Embedding> response = embeddingModel.embed(query);
        return embeddingStore.search(EmbeddingSearchRequest.builder()
                        .queryEmbedding(response.content())
                        .maxResults(defaultMaxResults)
                        .build())
                .matches();
    }
}

A few highlights:

• FileSystemDocumentLoader reads every file in a directory as a Document.

• DocumentSplitters.recursive(200, 20) chunks documents into segments of at most 200 characters with 20 characters of overlap, so meaning isn't cut off at chunk boundaries.

• EmbeddingStoreIngestor is the pipeline: split → embed → store. It returns token usage so you can see how much a document cost to embed.

• InMemoryEmbeddingStore supports serializeToFile / fromFile, so the store survives restarts. We load it in the constructor if the file already exists.

Switching Embedding Models

The issue's checklist asked us to try different embedding models, so we made the model switchable at runtime. In AiConfig we expose a small factory bean:

@Bean
public Function<String, EmbeddingModel> embeddingModelFactory() {
    return modelName -> OpenAiEmbeddingModel.builder()
            .apiKey(System.getenv("OPENAI_API_KEY"))
            .modelName(modelName)
            .build();
}

SemanticSearchService uses this factory to build its model, so setEmbeddingModel("text-embedding-3-large") swaps it on the fly. We support the three OpenAI embedding models:

text-embedding-3-small | text-embedding-3-large | text-embedding-ada-002

Note that switching models changes the vector dimensions, so re-index your documents after a switch — that's why the CLI reminds you.

Trying It Out

The CLI (our ChatCli from the memory post) gained embedding commands alongside chat. Index the bundled sample documents and search:

/index sample-data
Indexed 4 segment(s). Store now holds 4 embedding(s).

/search what is a vector database?
Top 5 results for: "what is a vector database?"
1. [score 0.5678] Retrieval Augmented Generation (RAG) combines a vector database with a chat model. ...
2. [score 0.4321] Embeddings are dense vector representations of text that capture semantic meaning. ...

The query never contains the exact phrase "vector database" from the top hit's first words, yet the model ranked it first because the meaning matches.

You can also inspect a raw vector:

/embed hello world
Embedding (1536 dimensions) of "hello world":
[0.0051, -0.0123, 0.0098, ...]

Configuration

Next Steps

Embeddings are the foundation of Retrieval Augmented Generation (RAG): retrieve the most relevant chunks for a question, then feed them to the chat model so it can answer from your own documents. That's a natural next exploration — and it builds directly on the memory, chat, and embedding pieces we now have.

Resources

• Official LangChain4j Documentation

• GitHub Examples Repository

• Embeddings Issue

Conclusion

Embeddings turn unstructured text into geometry, and semantic search is just a nearest-neighbor query over that geometry. With LangChain4j's EmbeddingStoreIngestor and InMemoryEmbeddingStore, the whole indexing pipeline is a few lines of code — and the store even persists to a plain JSON file. Combined with conversation memory, we're one step away from a full RAG application.

The full implementation lives in our GitHub repository. Stay tuned for RAG!

Why Does a Chatbot Need Memory?

Large language models are stateless. Each request is processed independently, and the model has no idea what you asked it one minute ago. To hold a real conversation, the application must keep the history and send it along with every new question.

This is exactly what LangChain4j's memory module does. It stores the conversation, decides what to include on each call, and automatically manages the size of what's sent so you stay within the model's context window.

Two Memory Strategies

LangChain4j 1.18.1 ships two built-in ChatMemory implementations, each controlling the sliding window differently:

1. MessageWindowChatMemory — a buffer limited by the number of messages. When the window is full, the oldest messages are evicted.

2. TokenWindowChatMemory — a window limited by the token budget. Messages are evicted until the history fits within a configured token count, using an actual tokenizer (in our case, OpenAI's).

Note: Older LangChain4j versions listed "Summary" and "Vector" memory types. These are no longer part of the current API — the two sliding-window strategies above are what's available today, and they map nicely to the classic "buffer" and "context-window" ideas.

Setup

We start from the getting-started project and add the core langchain4j module, which contains AiServices and the memory implementations:

<dependency>
    <groupId>dev.langchain4j</groupId>
    <artifactId>langchain4j-open-ai</artifactId>
</dependency>

<!-- Adds AiServices, MessageWindowChatMemory, TokenWindowChatMemory, ... -->
<dependency>
    <groupId>dev.langchain4j</groupId>
    <artifactId>langchain4j</artifactId>
</dependency>

Defining an AI Service

The idiomatic way to add memory in LangChain4j is through an AI Service — a plain Java interface that AiServices turns into a working implementation.

public interface Assistant {

    @SystemMessage("You are a helpful assistant. Answer the question in a very concise way, only in 2 sentences maximum.")
    String chat(@MemoryId String memoryId, @UserMessage String message);
}

The annotations tell LangChain4j everything it needs:

• @SystemMessage — the system prompt injected on every call.

• @MemoryId — selects which conversation's memory to use, so you can have one memory per user or per chat.

• @UserMessage — marks the parameter holding the user's input.

Wiring It Up with Spring

Next we define the beans: a ChatModel, and the Assistant built with a ChatMemoryProvider. The provider is called by AiServices the first time a new memory ID is seen, so we can decide which memory type to create based on the ID itself. This is the trick that lets us switch memory types at runtime — the memory type is embedded in the memory ID.

@Configuration
public class AiConfig {

    @Bean
    public ChatModel chatModel(@Value("${app.chat.model-name:gpt-4o-mini}") String modelName) {
        return OpenAiChatModel.builder()
                .apiKey(System.getenv("OPENAI_API_KEY"))
                .modelName(modelName)
                .build();
    }

    @Bean
    public Assistant assistant(ChatModel chatModel,
                               ChatMemoryRegistry chatMemoryRegistry,
                               @Value("${app.chat.model-name:gpt-4o-mini}") String modelName,
                               @Value("${app.memory.max-messages:10}") int maxMessages,
                               @Value("${app.memory.max-tokens:2000}") int maxTokens) {
        ChatMemoryProvider chatMemoryProvider = memoryId -> {
            ChatMemory chatMemory = createMemory((String) memoryId, modelName, maxMessages, maxTokens);
            chatMemoryRegistry.register((String) memoryId, chatMemory);
            return chatMemory;
        };

        return AiServices.builder(Assistant.class)
                .chatModel(chatModel)
                .chatMemoryProvider(chatMemoryProvider)
                .build();
    }

    private ChatMemory createMemory(String memoryId, String modelName, int maxMessages, int maxTokens) {
        if (memoryId.startsWith(MemoryType.MESSAGE_WINDOW.label())) {
            return MessageWindowChatMemory.builder()
                    .id(memoryId)
                    .maxMessages(maxMessages)
                    .build();
        }
        return TokenWindowChatMemory.builder()
                .id(memoryId)
                .maxTokens(maxTokens, new OpenAiTokenCountEstimator(modelName))
                .build();
    }
}

The MemoryType enum encodes each strategy with a label and produces the memory ID:

public enum MemoryType {

    MESSAGE_WINDOW("message-window"),
    TOKEN_WINDOW("token-window");

    // ...

    public String memoryId(String conversationId) {
        return label + ":" + conversationId;
    }
}

So a conversation id of main becomes either message-window:main or token-window:main. When you switch strategies, the ID changes, AiServices sees a new memory ID, and the provider hands back a fresh memory of the new type. Switching types effectively starts a new conversation — which is exactly the behavior you'd want.

AiServices retains the created memories itself, but it doesn't expose them. To let the CLI inspect and clear memory, we keep a small registry:

@Component
public class ChatMemoryRegistry {

    private final ConcurrentMap<String, ChatMemory> memories = new ConcurrentHashMap<>();

    public void register(String memoryId, ChatMemory memory) {
        memories.put(memoryId, memory);
    }

    public ChatMemory get(String memoryId) {
        return memories.get(memoryId);
    }
}

The Command-Line Interface

We moved the interactive loop out of the main application class into a dedicated ChatCli component, gated by a property so tests can load the Spring context without blocking on stdin.

app.cli.enabled=true
app.chat.model-name=gpt-4o-mini
app.memory.max-messages=10
app.memory.max-tokens=2000

The CLI now understands a few commands alongside plain chat:

/help                 Show this help
/memory               Show current memory type and state
/memory <type>        Switch memory type (message-window | token-window)
/clear                Clear the current conversation memory
quit                  Exit the application

Try it: ask "My name is Alice", then follow up with "What is my name?". With memory enabled, the model remembers. Then run /memory token-window and notice the conversation history was reset — a new memory of the new type starts fresh.

Running the Application

1. Set your API key: export OPENAI_API_KEY=...

2. Run it: ./mvnw spring-boot:run

3. Chat, and use /memory and /clear to explore how each memory type behaves.

Next Steps

Conversation memory is the foundation for much more advanced features:

1. RAG (Retrieval Augmented Generation) — combine memory with document retrieval

2. Chains and Agents — let the model use tools, with full conversation context

3. Embeddings — semantic search over past conversations

4. Streaming responses — stream tokens back to the user

Resources

• Official LangChain4j Documentation

• GitHub Examples Repository

• Conversation Memory Issue

Conclusion

Adding memory transforms a stateless question-answer loop into a real conversation. With LangChain4j's AI Services and @MemoryId, it takes only a few lines of code, and the built-in MessageWindowChatMemory and TokenWindowChatMemory give you two ways to manage the context window — switchable at runtime.

Check out the full implementation in our GitHub repository. Stay tuned for more explorations — RAG, agents, and tool use are next on the list!

In this post, I'll share how we solved both using a Generic Specification Repository and Slice-based Pagination.

1. The Problem: Boilerplate & Heavy Queries

The Boilerplate Trap

Typically, allowing users to filter by name, status, createdDate, etc., requires writing endless custom repository methods or complex CriteriaBuilder logic.

The Count Query Killer

Standard pagination (Page<T>) executing findAll(Pageable) triggers two queries:

1. The actual data query (SELECT ... LIMIT ? OFFSET ?).

2. A total count query (SELECT COUNT(*) ...).

For large tables, the count query is a performance killer. If you're building an "Infinite Scroll" or "Load More" feature, you don't even need the total count—you just need to know if there's a "next page".

2. The Solution: Generic Specification Repository

We implemented a wrapper around JpaSpecificationExecutor that accepts a dynamic SearchRequest. This allows us to filter any entity without writing custom query code.

The Interface

@NoRepositoryBean
public interface SpecificationRepository<T, ID> extends JpaRepository<T, ID>, SliceSpecificationExecutor<T> {    
    // Inherits findAll(Specification<T>, Pageable)    
    // Inherits findAllSliced(Specification<T>, Pageable)
}

The Usage

Now, any repository can extend this and instantly gain dynamic search powers:

public interface ThreatActorRepository extends SpecificationRepository<ThreatActor, UUID> {}

The Service layer simply builds the specification from a JSON request:

public Page<ThreatActor> search(SearchRequest request) {    
    Specification<ThreatActor> spec = GenericSpecificationBuilder.buildFromRequest(request);    
    return repository.findAll(spec, pageable);
}

3. The Optimization: Slice vs. Page

To solve the count query performance issue, we implemented SliceSpecificationExecutor.

What is a Slice?

A Slice in Spring Data holds a chunk of data and knows if there is a next slice (hasNext()), but it does not know the total number of pages.

Implementing findAllSliced

We leveraged Spring Data's Window API (introduced recently) to fetch results efficiently:

SliceSpecificationExecutor.javadefault Slice<T> findAllSliced(Specification<T> spec, Pageable pageable) {    // Uses a "Window" to fetch size + 1 items to determine if a next page exists    // completely avoiding the SELECT COUNT(*) query.    Window<T> window = this.findBy(spec, ...);    return new SliceImpl<>(window.getContent(), pageable, window.hasNext());}

The Performance Win

By switching from /search (Page) to /search-sliced (Slice), we eliminate the heavy count query entirely.

SQL Comparison:

Regular Page Request:

sqlHibernate: select ... from threat_actors limit ? offset ?Hibernate: select count(*) from threat_actors ... -- 🛑 EXPENSIVE on large tables

Slice Request:

sqlHibernate: select ... from threat_actors limit ? offset ? -- ✅ ONLY ONE QUERY

4. Conclusion

By combining the Specification Pattern with Slice Pagination, we achieved:

1. Cleaner Code: Zero boilerplate for dynamic filters.

2. Better Performance: 50% fewer queries for infinite scroll views.

Check out the full implementation in the repository!

References

• https://github.com/prasadgaikwad/spring-data-jpa-specification

• https://gist.github.com/josergdev/06c82891a719eca4834410339885ad23

• Spring Data JPA Specifications

• Vlad Mihalcea: The best way to use the Spring Data JPA Specification

What is LangChain4j?

LangChain4j is a powerful Java framework designed to simplify the development of applications powered by Large Language Models (LLMs). It provides a comprehensive set of tools and abstractions that make it easier to build sophisticated AI-powered applications.

Project Setup

Prerequisites

• Java 17 or higher

• Maven

• Your favorite IDE (IntelliJ IDEA, Eclipse, or VS Code)

• An API key from your chosen LLM provider (e.g., OpenAI)

Step 1: Create a Spring Boot Project

Start by creating a new Spring Boot project using Spring Initializr or your IDE. We'll need the following dependencies:

• Spring Boot Starter

• LangChain4j BOM (Bill of Materials)

Here's our pom.xml configuration:

<!-- Parent Spring Boot -->
<parent>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-parent</artifactId>
    <version>3.5.7</version>
</parent>

<!-- LangChain4j BOM for version management -->
<dependencyManagement>
    <dependencies>
        <dependency>
            <groupId>dev.langchain4j</groupId>
            <artifactId>langchain4j-bom</artifactId>
            <version>1.8.0</version>
            <type>pom</type>
            <scope>import</scope>
        </dependency>
    </dependencies>
</dependencyManagement>

<dependencies>
    <!-- Spring Boot Starter -->
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter</artifactId>
    </dependency>

    <!-- LangChain4j OpenAI Integration -->
    <dependency>
        <groupId>dev.langchain4j</groupId>
        <artifactId>langchain4j-open-ai</artifactId>
    </dependency>
</dependencies>

Step 2: Configuration

Create or update your .env file: (make sure to add this .env file to your .gitignore to keep your API key secure)

# OpenAI API Configuration
OPENAI_API_KEY=your-api-key-here

Step 3: Creating the Application

We'll create a simple command-line application that:

1. Asks for user's question

2. Answers concisely using the OpenAI GPT-4o-mini model

3. Continues interaction until they type 'quit'

The implementation uses Spring's CommandLineRunner for the interactive loop.

Understanding the Code

Let's break down the key components:

1. The Main Application Class

@SpringBootApplication
public class LangChain4jDemoApplication implements CommandLineRunner {

    public static void main(String[] args) {
        SpringApplication.run(LangChain4jDemoApplication.class, args);
    }

    /**
     * A simple command-line application that interacts with the user,
     * takes a question as input, and provides a concise answer using
     * the OpenAI GPT-4o-mini model.
     */
    @Override
    public void run(String... args) {
        ChatModel model = OpenAiChatModel.builder()
                .apiKey(System.getenv("OPENAI_API_KEY"))
                .modelName("gpt-4o-mini")
                .build();

        try (Scanner scanner = new Scanner(System.in)) {
            while (true) {
                System.out.print("Please enter your question (type 'quit' to exit): ");
                String question = scanner.nextLine();

                if ("quit".equalsIgnoreCase(question)) {
                    System.out.println("Goodbye!");
                    break;
                }

                String response = model.chat("You are an helpful assistant. " +
                        "Answer this question in very concise way, only in 2 sentences maximum. Question: " + question);
                System.out.println("Answer: " + response);
                System.out.println(); // Add a blank line for better readability
            }
        }
    }
}

Running the Application

To run the application:

1. Ensure you have set your API key in .env file

2. Run the following command:

    ./mvnw spring-boot:run
   

Next Steps

This basic setup provides a foundation for exploring more advanced LangChain4j features:

1. Adding LLM Integration

   • Implement chat completions

   • Add streaming responses

   • Experiment with different models

2. Implementing Memory

   • Add conversation history

   • Implement different memory types

3. Creating Chains

   • Build processing pipelines

   • Add prompt templates

Resources

For more information and advanced features, check out:

• Official LangChain4j Documentation

• GitHub Examples Repository

Conclusion

This simple example demonstrates how easy it is to get started with LangChain4j. The framework's integration with Spring Boot makes it particularly attractive for Java developers looking to build AI-powered applications.

Stay tuned for more blog posts where we'll explore advanced features like:

• Working with different LLM providers

• Implementing RAG (Retrieval Augmented Generation)

• Building custom agents and tools

• Creating sophisticated conversation chains

Remember to check out our GitHub repository for the complete source code and future updates!

Note: The steps mentioned here are specifically for Google DNS, a .dev domain and Hashnode.

So I wanted to create my blog and online profile for quite some time but was not sure about following:

1. How to start the blog?
2. How to register a domain name and setup?
3. Should I use any blogging platform or build my own blog?
4. Which blogging platform to use?

And after researching for a while, I decided to create my blog today to check this item off my list on very first day of the year :)

Here is a short tutorial on how to create a blog, setup your own domain name and use the domain name to redirect to your blog, all about in 10 minutes. (off course it took some time for me to research on the options for blogging platforms, domain name service providers etc, but actual process of blog setup took about 10 minutes!)

My Choices

• I chose Google Domains as my domain name service (DNS) provider to avoid getting too much into the weeds for searching a DNS provider. During my initial research, I found it has features which I required to get started quickly.
• I picked Hashnode as my blogging platform since I found that it provides a lot of useful features in addition to the ease of creating and maintaining a blog.

Prerequisites

1. You need a google/gmail account to buy domain name from google.
2. Create an account with Hashnode to get you started (you don't need steps for this, believe me ;))

Steps

1. Search for your required domain name at Google Domains, add it to the cart and proceed. You should see something like following on next screen:
2. Provide required contact information on checkout screen.
3. Provide payment information and checkout. That's it! You own the domain name now! You may have to verify your email address for the domain purchase.
4. On your Hashnode blog dashboard page, add your domain name (in my case I used a subdomain blog.prasadgaikwad.dev) under DOMAIN tab and click update. You will see next steps on the same page as shown below:
5. Go back to your Google Domains dashboard and select the newly created domain name to go into its settings.
6. Go to DNS section and scroll to the "Custom resource records" section. Add CNAME record as shown below: (Note that I am using a subdomain "blog" since Google Domains does not support CNAME record for root domain. You can create any subdomain)
7. Wait for few minutes and go to your subdomain url, you should see your Hashnode blog!
8. Wait for it...you may also want to redirect your root domain (in my case prasadgaikwad.dev and www.prasadgaikwad.dev), then go the Website section of your Google Domains page and add Forward domain details. (In my case I forwarded my root domain to my blog subdomain for now!)
9. You may have to hard refresh (command + r) browser page with your domain url, to delete any cached pages or errors.
10. And that's all you need to get your own developer blog started on your custom domain.

References:

1. How to Set Up a Custom Domain on Devblog
2. Hashnode: How to Launch Your Own Developer Blog on Your Own Domain in Minutes

(I created this blog post to document steps which I could not find in reference blogs, so that it might be useful if someone decides to use these platforms like me)