--- title: Evaluate an MCP server description: >- Give the agent your MCP server, check that it calls the server's tools, and measure whether answers improve. --- > ## Documentation Index > Fetch the complete documentation index at: https://docs.reeval.ai/llms.txt > Use this file to discover all available pages before exploring further. **Goal:** find out whether agents call your MCP server, and whether they answer better with it than without. The server goes in an [environment](/guide/environments), and a `compare` runs the eval with and without it. `toHaveCalledMcp(server, tool)` checks the call, which the transcript names `mcp____`. ## The files ```ts title="evals/environments.ts" import { defineEnvironment } from '{{ pkg }}' export const plain = defineEnvironment('plain', {}) export const withDocsMcp = plain.extendEnvironment( 'with-docs-mcp', ({ mcp, secret }) => ({ mcp: { docs: mcp.http('https://mcp.acme.dev/mcp', { headers: { Authorization: secret.template('Bearer ${ACME_TOKEN}'), }, }), }, }), ) ``` ```ts title="evals/docs-mcp.eval.ts" import { evaluate, expect, group } from '{{ pkg }}' import { plain, withDocsMcp } from './environments.ts' group( 'acme docs MCP', { compare: { baseline: plain, variants: [withDocsMcp] }, runs: 3, gate: { passRate: 0.6 }, }, () => { evaluate('looks up the latest SDK version', async (c) => { const turn = await c.send( 'Which version of acme-sdk is the latest? ' + 'Reply with the version only.', ) if (c.attempt.cell.role === 'variant') { expect(turn).toHaveCalledMcp('docs', 'latest_version') } expect(turn.text).toContain('4.2.0') }) }, ) ``` - `mcp.http(url, { headers })` connects the agent to a remote server. `mcp.sse(url)` works the same way for SSE servers, except on OpenCode, Goose, pi and Codex, which have no SSE transport, so their plans exit 2. - `secret.template('Bearer ${ACME_TOKEN}')` reads `ACME_TOKEN` from the host when the attempt starts. - The MCP check runs only in the variant cell: the control has no server to call. Copy-paste from a transcript is `toHaveCalledTool('mcp__docs__latest_version')`. ## The token stays out of the sandbox The URL is `https://`, so reeval brokers the header: the agent's MCP config holds a placeholder, and the egress gateway swaps in the real token only on requests to `mcp.acme.dev`. The plan says so: ```bash npx reeval run --dry-run ``` ```text plan 1 eval · 2 cells · 6 attempts · ~18m 00s serial (~4m 30s at concurrency 4) eval harnesses variants runs attempts notes evals/docs-mcp.eval.ts › acme docs MCP › looks up the latest SDK… 1 1 3 6 brokered: mcp.docs.headers.Authorization → mcp.acme.dev ``` A restricted [egress policy](/concepts/egress) allows the server's host automatically, so the variant still reaches it under `egress: 'deny-all'`. An unset `ACME_TOKEN` doesn't stop the plan. The variant's attempts fail with `secret.template for … needs ACME_TOKEN, which isn't set. Set it in the environment or .env.local.`. Put it in `.env.local` next to your harness key. ## Run it ```bash npx reeval image build npx reeval run docs-mcp ``` Each variant attempt's transcript names the server and tool, the same way on every harness: ```text {"turn":0,"ts":"2026-09-26T17:45:18.116Z","atMs":0,"type":"tool","toolCallId":"call_1","tool":"mcp__docs__latest_version","input":{},"output":"acme-sdk 4.2.0","ok":true,"durationMs":112} ``` That line comes from a fake run (below). In the studio, the attempt's Transcript tab shows the same call, and `reeval run -v` prints a `harness tool mcp__docs__latest_version called` line as it happens. `toHaveCalledMcp('docs', 'latest_version')` is the same fact. ## Try it without keys The [fake harness](/recipes/fake-harness) can call a stdio MCP server's tool, so you can check the wiring before spending credit. Point a second environment at a small local server: ```ts title="evals/environments.ts" export const withLocalDocs = defineEnvironment( 'with-local-docs', ({ file, mcp }) => ({ files: { 'mcp/server.mjs': file('../mcp/server.mjs') }, mcp: { docs: mcp.stdio('node', ['mcp/server.mjs']) }, }), ) ``` ```ts title="reeval.fake.config.ts" import { defineConfig, defineEnvironment } from '{{ pkg }}' import { fakeHarness } from '{{ pkg }}/harness' import { fakeSandbox } from '{{ pkg }}/sandbox' export default defineConfig({ include: ['evals/local-mcp.eval.ts'], harness: fakeHarness({ script: [[{ mcp: 'docs/latest_version' }, { reply: '4.2.0' }]], }), environment: defineEnvironment('fake', { sandbox: fakeSandbox() }), }) ``` `evals/local-mcp.eval.ts` is the eval above with `withLocalDocs` in place of `withDocsMcp`. `mcp/server.mjs` is any stdio MCP server with a `latest_version` tool: ```js title="mcp/server.mjs" import { createInterface } from 'node:readline' const reply = (id, result) => process.stdout.write(`${JSON.stringify({ jsonrpc: '2.0', id, result })}\n`) createInterface({ input: process.stdin }).on('line', (line) => { const { id, method } = JSON.parse(line) if (method === 'initialize') { reply(id, { protocolVersion: '2024-11-05', capabilities: { tools: {} }, serverInfo: { name: 'docs', version: '1.0.0' }, }) } else if (method === 'tools/list') { reply(id, { tools: [{ name: 'latest_version', inputSchema: { type: 'object' } }], }) } else if (method === 'tools/call') { reply(id, { content: [{ type: 'text', text: 'acme-sdk 4.2.0' }] }) } else if (id !== undefined) { // Any other request gets "method not found", or the client waits for it. const error = { code: -32601, message: `Method not found: ${method}` } process.stdout.write(`${JSON.stringify({ jsonrpc: '2.0', id, error })}\n`) } }) ``` Answer every request, even ones your server doesn't support. Goose probes `server/discover` before `initialize`; a server that stays silent adds its 10-second timeout to every turn. Run only the variant, since the control has no server for the script to call: ```bash npx reeval run --config reeval.fake.config.ts with-local-docs --output-logs full ``` ```text [looks#1 with-local-docs] ▶ evals/local-mcp.eval.ts › acme docs MCP › looks up the latest SDK version … [looks#1 with-local-docs] +0:02 sandbox ready 2s [looks#1 with-local-docs] +0:02 turn 1 started [looks#1 with-local-docs] +0:02 turn 1 done 177ms · 1 tool call, 0 file edits … pass acme docs MCP › looks up the latest SDK version arm pass with-local-docs 3 of 3 can't tell yet 4s · results .reeval/runs/20261020T214302Z_…/summary.md 1 pass ``` With the control filtered out, the variant has nothing to be measured against, so its effect reads `can't tell yet`. ## Variations | Variation | How | |---|---| | A published stdio server | `mcp.stdio('npx', ['-y', '@acme/docs-mcp@1.0.0'])`. Pin the version, and give its env secrets `hosts` to broker them: `secret.fromEnv('ACME_TOKEN', { hosts: ['api.acme.dev'] })` | | Check the answer, not the call | Drop the tool check and add `c.judge('Does the reply name 4.2.0 as the latest version?')` | | Server plus docs skill | `compare: { baseline: plain, variants: [withDocsMcp, withDocsSkill] }` compares both against the same control, and `withDocsMcp.extendEnvironment('mcp-and-skill', …)` adds a cell with both | | Every harness | Set `harness: [cursor(…), claudeCode(…)]`. `toHaveCalledMcp` and the `mcp____` transcript name are the same on each (see [Compare agents](/recipes/compare-agents)), except that pi writes a `-` in a tool's name as `_` ([pi](/reference/harness/pi)) | ## Next How the agent uses a token it never sees. Gate on the lift with `minLift`.