Evals
An eval is a task in your users' words ("Is the login bug fixed yet?"), given to an agent that has only your server's tools. The agent calls your server for real, and a judge reads every call and the final answer to decide whether the task was done. When it wasn't, it says why and whether the cause is your server (a misleading description, an unclear parameter, an error that doesn't say how to recover, a result that's too big) or the agent, with a concrete fix.
Evals are on the server's Evals page, under the server in the sidebar.
Give evals the server's URL
Evals connect to your server over Streamable HTTP. Set its Server URL on the server's card under Project settings → Setup → MCP servers. If the server needs a key, add it under Headers to send (stored encrypted). The SDK can stay where it is: eval traffic is left out of your analytics.
Add evals
Suggest evals drafts a few from your tools and the last month of sessions: the tasks agents really come to do, and the ones they fail at. Keep the ones that matter, or write your own with New eval. On any session of the server, Turn into an eval drafts one from what that agent was trying to do, so you can try it again after a fix.
Run them
Run all, or Run now on one eval. Each agent (Claude, ChatGPT) runs the task once, in a minute or two. Evals can also run every day, or whenever the server's tool list changes.
What an eval checks
| Field | |
|---|---|
| The agent is asked | The task, in a user's words, with the ids or names a user would give |
| A good result | One point a line. The judge marks each met or not |
| Must call / must not call | Tools the agent has to call (optionally in order), and tools it mustn't, like a delete in a read-only task |
| Most tool calls | The budget: going over fails the eval |
| Pass score | Out of 100, from the judge (default 70) |
| Decoy tools | Adds a dozen unrelated tools (web search, files, email…) next to yours, to see whether the agent still picks yours from names and descriptions alone |
A run passes when every check passes and the judge's score reaches the pass score. It's inconclusive when it couldn't run: the server didn't answer, or refused sign-in.
A run's page
- Problems, each attributed to your server, the agent, or the setup, with the fix. Copy prompt for your coding agent gives Claude Code or Cursor what it needs to make the change.
- Tool calls in order, with the arguments the agent sent, what came back and how many tokens it was.
- Final answer, and what the run cost in AI tokens.
Test a fix before you ship it
The AI review on Recommendations suggests new tool definitions. Test with evals runs the server's evals twice, with the definitions as they are and with the suggested ones, so you see whether the new description actually helps. Agents read the new definitions; the server still runs your current code, so a fix that changes parameters needs deploying to test fully.
From your coding agent or CI
The MCP server has list_evals, run_evals and get_eval_run, so a coding agent can change a tool, deploy, and check its evals still pass. In CI, after a deploy:
curl -X POST https://agentstats.io/api/v1/sites/<site id>/evals/run \
-H "Authorization: Bearer al_sk_…" -H "Content-Type: application/json" \
-d '{ "source": "<MCP server id>" }'
# → { "runs": [{ "id": "run_…", "status": "queued" }, …] }
curl https://agentstats.io/api/v1/sites/<site id>/evals/runs/run_… -H "Authorization: Bearer al_sk_…"
# → { "status": "done", "verdict": "passed", "score": 86, "problems": [ … ] }
Limits
Each run is one agent's attempt at one eval, judged. Plans include 10 (Hobby), 60 (Starter), 300 (Growth) or 1,500 (Scale) runs a month for the whole team. A run usually costs a few cents of AI tokens, which is why there's a cap.