Advanced Features: Multi-Modal, Function Calling, and Structured Output

The demo has covered memory, RAG, agents, prompting, integration, tooling, and evaluation. This round attacks the checklist's advanced features item: multi-modal capabilities, function calling, structured output, and custom tools. The pattern that worked in earlier rounds holds: every capability is exercised offline in tests using fake models, while the real OpenAI models are wired for runtime use.

Multi-Modal

"Multi-modal" here means three separate capabilities, each backed by a different model interface in LangChain4j.

Vision: images into the chat model

A multimodal chat model (GPT-4o-mini) accepts image content alongside text. VisionService builds a UserMessage that mixes a TextContent (the question) with an ImageContent:

UserMessage.from(TextContent.from(question), ImageContent.from(URI.create(imageUrl)))

The image can come from a public URL or, more usefully, as raw bytes — the service base64-encodes them and passes a mime type, which is how an uploaded file reaches the model:

ImageContent.from(Base64.getEncoder().encodeToString(imageData), mimeType)

POST /api/describe exposes this over HTTP ({imageUrl | imageData, mimeType, question}), and /describe <url> [question] works from the CLI.

Image generation

ImageGenerationService wraps an ImageModel — OpenAI's gpt-image-1 by default, configurable via app.image.model-name. The returned Image carries a URL, or base64 data plus a mime type, or a revisedPrompt. The CLI /generate <prompt> prints whichever form came back.

Speech-to-text

SpeechToTextService wraps an AudioTranscriptionModel (whisper-1). It wraps raw audio bytes into an Audio value and calls transcribeToText:

Audio audio = Audio.builder()
        .base64Data(Base64.getEncoder().encodeToString(audioData))
        .mimeType(mimeType)
        .build();
return transcriptionModel.transcribeToText(audio);

CLI: /transcribe <file>. The wav/mp3/ogg/m4a extension is mapped to a mime type so the model knows what it's parsing.

Function Calling and Custom Tools

The agent from an earlier round already calls @Tool methods. This round pushes further on what a tool can be.

Structured tool parameters

WeatherTool.getWeather takes a WeatherRequest record instead of a handful of scalars:

@Tool("Gets the current temperature for a city, returning it in the requested unit")
public String getWeather(WeatherRequest request) { ... }

@Description("Weather request parameters")
public record WeatherRequest(String city, TemperatureUnit unit) { }

public enum TemperatureUnit { CELSIUS, FAHRENHEIT }

LangChain4j derives a nested object schema from the record — city as a string, unit as an enum — so the model can fill in both fields in one call. The weather data is fake and deterministic, so the tool works offline. One subtlety surfaced in testing: for a POJO parameter named request, the model's arguments must be {"request": {"city": "...", "unit": "..."}}, keyed by the parameter name, not the bare object.

Conversation-scoped tool state

NoteTool demonstrates a tool with state that is aware of which conversation it is running in, via @ToolMemoryId:

@Tool("Saves a note for the current conversation")
public String saveNote(@ToolMemoryId String memoryId, @P("The note text to save") String note) { ... }

LangChain4j injects the current conversation's memory id into the call, so notes saved in one conversation never leak into another — a real pattern for memory-adjacent tools.

Dynamic tool selection with a ToolProvider

The static agent registers its tools at build time with .tools(...). The new DynamicToolProvider instead decides per request which tools are available, using ToolProviderRequest to inspect the user message:

@Override
public ToolProviderResult provideTools(ToolProviderRequest request) {
    List<AiServiceTool> tools = new ArrayList<>(ToolService.findTools(calculatorTool));
    tools.addAll(ToolService.findTools(noteTool));
    if (request.userMessage().singleText().toLowerCase().contains("weather")) {
        tools.addAll(ToolService.findTools(weatherTool));
    }
    return ToolProviderResult.builder().addAll(tools).build();
}

The calculator and note tools are always exposed; the weather tool only appears when the task is actually about the weather. That keeps the model's function-call surface — and the tokens describing it — as small as possible. AiServices.builder(DynamicAgent.class).toolProvider(provider) wires it up, and /dynamic <task> exercises it from the CLI.

Structured Output at the Model Level

The demo already had structured output through AI Service return types (MovieReview, enums, List<String>), where LangChain4j asks for JSON in the prompt and parses the reply. This round adds the model-level approach: attach the JSON schema derived from the record to the request as a response format, constraining the model to emit exactly that shape:

JsonSchema schema = JsonSchemas.jsonSchemaFrom(MovieReview.class)
        .orElseThrow(...);

ChatRequest request = ChatRequest.builder()
        .messages(List.of(SystemMessage.from(SYSTEM_PROMPT), UserMessage.from(text)))
        .parameters(ChatRequestParameters.builder()
                .responseFormat(schema)   // model is constrained to this schema
                .build())
        .build();

/schema <text> runs this from the CLI. The guarantee is stronger than prompt-based extraction: with JSON mode the model's output is structurally enforced rather than merely requested.

Testing It All Offline

The suite grew to 92 tests, all offline:

• Vision — a FakeChatModel captures the built ChatRequest; tests assert the user message contains exactly one TextContent and one ImageContent, and that the base64 variant carries the right mime type.

• Image generation / STT — dedicated fakes (FakeImageModel, FakeAudioTranscriptionModel) capture the prompt and the Audio payload respectively.

• Custom tools — direct unit tests for WeatherTool (unit conversion, unknown city) and NoteTool (notes scoped per conversation).

• Function calling end-to-end — DynamicToolProviderTest builds a real DynamicAgent over AiServices with a fake chat model that returns a ToolExecutionRequest on its first call and a plain answer on its second. The test watches the tool-specification list on each round: the weather tool is present only when the task mentions weather, the tool actually executes, and its result flows back into the second model call.

• Structured output — asserts the request's responseFormat().jsonSchema() is named MovieReview and that the model's JSON reply parses into the record.

The agent-loop test deserves emphasis: it exercises the real AiServices machinery — schema generation, tool dispatch, the round-trip of results back into the model — with zero network calls, purely by scripting what the model would say.

Design Notes

• Fake models everywhere. No capability ships without an offline test proving the request wiring is correct.

• Runtime configurability. Image and STT models are beans you can swap by property, matching the existing chat/embedding model pattern.

• Dynamic > static where it matters. Exposing a tool per request beats registering every tool for every task — smaller prompts, fewer wrong calls.

• Nested tool schemas. A record parameter is the clean way to give a tool rich, typed inputs; just remember the arguments are keyed by the parameter name.

Next Steps

The last checklist item is LLM integration: wiring alternative providers (Anthropic, Google, Ollama) and comparing models. The /eval harness from the previous round is exactly the yardstick that comparison needs.

Resources

• LangChain4j Tools Documentation

• LangChain4j Multi-Modality

• Advanced Features Issue

Conclusion

The advanced-features round turned the demo from a text-only assistant into one that sees images, paints them, listens, calls tools with structured parameters, and emits schema-constrained JSON. Two ideas carried the round: fakes make even a tool-calling agent loop testable offline, and the interface-based model abstraction in LangChain4j (chat, image, audio) means each new modality was a small service rather than a rewrite.

The full implementation lives in our GitHub repository.