Blog

Making the JProfiler MCP Server Robust for Weaker Models

Using flagship models is not ideal for all coding tasks. They are slow and they cost a lot. So many tasks, especially chores, are more suitable for lower-tier models that are more nimble and do the task at a fraction of the cost. However, the tool-calling behavior of these models can be more erratic than what the top models are capable of.

In JProfiler 16.1 we added an MCP server to let AI coding agents profile Java applications and analyze the results. This has been met with a lot of positive feedback from our customers, so it is an area that we intend to expand on in the upcoming releases.

For the initial release, most of our testing had been done mainly with Claude Code. For JProfiler 16.2, we have made the MCP server a lot more robust for other AI models. Read on for what the other models did wrong and how we made the profiling workflow robust against their shortcomings.

For impatient readers, here's the result in one chart: four models running our eleven profiling scenarios against both the 16.1.1 and the 16.2 server.

Tool Descriptions Are Only Suggestions

An MCP server is an API whose caller is a language model. With a normal API, the caller either follows the contract or gets an error. A language model reads tool descriptions as suggestions and improvises when things are unclear or there is no good match with its goal. The stronger the model, the less this happens. When we tested the 16.1 server with other models, we saw that the profiling workflow failed in several different ways:

• Instead of adding the prepared JVM parameter for profiling to the JVM invocation, models created their own configuration files or modified existing files, and the recording never started.
• After the test run, models loaded the saved JProfiler snapshot file directly instead of asking the MCP server for the session status, which bypasses the session management of the server.
• Because the internal configuration file of the MCP server appeared as a readable path in the JVM parameter, models inspected it, and some tried to modify it.

This is not specific to JProfiler. The τ-bench benchmark evaluates tool-using agents in simulated customer service domains and finds that even the best models succeed in less than half of the tasks and give inconsistent results when the same task is repeated.

Guardrails in the Tools

In 16.2, the workflow is hardened in many ways, for example:

• The tool descriptions are now prescriptive. The prepare_profiling tool tells the agent to add the JVM parameter "verbatim and do not create or modify any files", and after the run to call check_status and "do not load the saved snapshot file yourself".
• The tools enforce the workflow. The load_snapshot tool refuses to open internal recording files of the MCP server and explains the correct next step in its error message.
• The path of the internal configuration file in the JVM parameter is obfuscated, so models cannot read or overwrite the file even if they try.

The general principle is that error messages are part of the prompt. When a model does something wrong, the response it gets from the MCP server tells it what to do instead. In other words, a model that goes off the rails is steered back by the next tool response. The article on writing effective tools for agents by Anthropic tells this story in detail: tool descriptions are prompt engineering, and tools have to be evaluated with the agents that will use them.

Testing That Agents Follow the Workflow

How do you regression-test that a language model follows a workflow? We built end-to-end tests that give a natural language profiling task to a coding agent, for example "record this program from startup and tell me where the time goes". A test then asserts two things: the agent called the expected MCP tools, and its answer is based on the actual profiling data.

The scenarios cover the main profiling tasks:

• CPU hotspots in a snapshot and drilling into a hotspot to find its caller
• the slowest SQL statement in a snapshot with JDBC data
• comparing two snapshots to find the cause of a regression
• recording a program from startup with prepare_profiling
• attaching to a running JVM, by PID or via the JVM list
• heap dumps and finding a memory leak in an HPROF file

The same scenarios run against many models. With OpenCode and OpenRouter we test models including DeepSeek, Gemini, GLM, GPT, Grok, Kimi, Mistral and Qwen, and the native CLIs of Claude Code, Codex, Cursor and Antigravity each have their own test class.

We have found that such a rigorous testing harness is strictly required to uphold a sufficient quality level that makes the MCP server suitable for use with a wide range of models. The Berkeley Function Calling Leaderboard, the most widely used benchmark for tool calling, has made the same move in its latest version: from single tool calls to the agentic evaluation of complete tasks.

Measuring the Effects

To quantify the improvement for this blog post, we ran four of the tested models through all eleven scenarios, five times each, with the OpenCode CLI as the agent harness, once against the 16.1.1 server and once against the 16.2 server. Each scenario is scored by its assertions: the expected MCP tools in the expected order, and an answer grounded in the profiling data.

The most striking case is GPT. gpt-5.6-sol is one of the strongest models available today, and it was the model that struggled most with the 16.1.1 server. The problem was clearly not model intelligence but the fact that the 16.1 workflow was calibrated to the way that Claude reads tool descriptions. Other models improvise in different ways when instructions leave room for interpretation, so that workflow was not optimal for them.

On 16.1.1, GPT failed the same three scenarios in every single repetition. The transcripts show it calling load_snapshot ten times in a row instead of moving on in the workflow, or starting a new recording when it was supposed to analyze an existing snapshot. On 16.2, it passes all tests. DeepSeek and Qwen were already reliable most of the time on 16.1.1, and Kimi was reliable throughout. The one remaining failure on 16.2 is a single Qwen run that failed its scenario three times in a row.

The tool calling failures are expensive. In the 16.1.1 runs, a failing GPT scenario took an average of 42 tool calls and 4 minutes before giving up, about eight times the cost of a passing run. Across the whole GPT matrix, the 16.1.1 server cost $13.21 for a 69% pass rate and the 16.2 server cost $3.13 for a 100% pass rate.

Getting Started

Try using the JProfiler MCP server for a nightly routine profiling job to check the performance impact of your commits. It is available on the download page, and the MCP setup page gets your agent connected in a minute.