MCP servers

Evals

Give Claude and ChatGPT a task with only your MCP server's tools, run it against the real server, and see what to fix.

An eval is a task in your users' words ("Is the login bug fixed yet?"), given to an agent that has only your server's tools. The agent calls your server for real, and a judge reads every call and the final answer to decide whether the task was done. When it wasn't, it says why and whether the cause is your server (a misleading description, an unclear parameter, an error that doesn't say how to recover, a result that's too big) or the agent, with a concrete fix.

Evals are on the server's Evals page, under the server in the sidebar.

Give evals the server's URL

Evals connect to your server over Streamable HTTP. Set its Server URL on the server's card under Project settings → Setup → MCP servers. If the server needs a key, add it under Headers to send (stored encrypted). The SDK can stay where it is: eval traffic is left out of your analytics.

Add evals

Suggest evals drafts a few from your tools and the last month of sessions: the tasks agents really come to do, and the ones they fail at. Keep the ones that matter, or write your own with New eval. On any session of the server, Turn into an eval drafts one from what that agent was trying to do, so you can try it again after a fix.

Run them

Run all, or Run now on one eval. Each agent (Claude, ChatGPT) runs the task once, in a minute or two. Evals can also run every day, or whenever the server's tool list changes.

What an eval checks

Field
The agent is askedThe task, in a user's words, with the ids or names a user would give
A good resultOne point a line. The judge marks each met or not
Must call / must not callTools the agent has to call (optionally in order), and tools it mustn't, like a delete in a read-only task
Most tool callsThe budget: going over fails the eval
Pass scoreOut of 100, from the judge (default 70)
Decoy toolsAdds a dozen unrelated tools (web search, files, email…) next to yours, to see whether the agent still picks yours from names and descriptions alone

A run passes when every check passes and the judge's score reaches the pass score. It's inconclusive when it couldn't run: the server didn't answer, or refused sign-in.

A run's page

  • Problems, each attributed to your server, the agent, or the setup, with the fix. Copy prompt for your coding agent gives Claude Code or Cursor what it needs to make the change.
  • Tool calls in order, with the arguments the agent sent, what came back and how many tokens it was.
  • Final answer, and what the run cost in AI tokens.

Test a fix before you ship it

The AI review on Recommendations suggests new tool definitions. Test with evals runs the server's evals twice, with the definitions as they are and with the suggested ones, so you see whether the new description actually helps. Agents read the new definitions; the server still runs your current code, so a fix that changes parameters needs deploying to test fully.

From your coding agent or CI

The MCP server has list_evals, run_evals and get_eval_run, so a coding agent can change a tool, deploy, and check its evals still pass. In CI, after a deploy:

curl -X POST https://agentstats.io/api/v1/sites/<site id>/evals/run \
  -H "Authorization: Bearer al_sk_…" -H "Content-Type: application/json" \
  -d '{ "source": "<MCP server id>" }'
# → { "runs": [{ "id": "run_…", "status": "queued" }, …] }

curl https://agentstats.io/api/v1/sites/<site id>/evals/runs/run_… -H "Authorization: Bearer al_sk_…"
# → { "status": "done", "verdict": "passed", "score": 86, "problems": [ … ] }

Limits

Each run is one agent's attempt at one eval, judged. Plans include 10 (Hobby), 60 (Starter), 300 (Growth) or 1,500 (Scale) runs a month for the whole team. A run usually costs a few cents of AI tokens, which is why there's a cap.