--- title: Evals and checks description: >- Write evals with prompts, sandbox commands, file checks, matchers and an LLM judge. --- > ## Documentation Index > Fetch the complete documentation index at: https://docs.reeval.ai/llms.txt > Use this file to discover all available pages before exploring further. An eval gives the agent a task, then checks what it did. Here's the first `greet` eval again, the one you ran on Cursor: ```ts title="evals/hello.eval.ts" import { evaluate, expect, group } from '{{ pkg }}' group('greet module', () => { evaluate('creates a tested greet module', async (c) => { const turn = await c.send( 'Create greet.js exporting greet(name) that returns "Hello, !", ' + 'and greet.test.js that tests it with node:test. No dependencies.', ) await expect(turn).toHaveFile('greet.js') await expect(c.exec('node --test')).toSucceed() expect( await c.judge( "Does the agent's final message accurately describe what it changed?", ), ).toPass() }) }) ``` It has three parts: a prompt, deterministic checks, and a question for the judge. `examples/with-checks` in the reeval repository is checks on their own: the agent adds `multiply` to a small package, and the checks read its files and run its tests. It runs on a fake agent with no keys or Docker: `pnpm eval` there. ## Name evals with groups ```ts title="evals/hello.eval.ts" group('greet module', () => { evaluate('creates a tested greet module', async (c) => { /* … */ }) evaluate('adds a default name in a follow-up turn', async (c) => { /* … */ }) }) ``` `evaluate(name, fn)` declares one eval, and `group(name, fn)` groups evals. Groups nest, and both take options as a second argument (or last). The file, groups and name make the eval's id, `evals/hello.eval.ts › greet module › creates a tested greet module`, which is how reeval tracks it across runs. Renaming an eval or its group gives it a new id, so its history starts again. Name the eval after the question it answers. The run's insights read the name first to tell what the eval checks, then explain the result in those terms. | | Name | |---|---| | Good | `recommends Genkit for a production AI feature` | | Bad | `genkit test 2` | ## Send a prompt ```ts const turn = await c.send('Create greet.js exporting greet(name).') turn.text // the agent's final message turn.status // 'completed', 'aborted' or 'error' turn.diff.files // [{ path: 'greet.js', status: 'added' }] ``` `c.send` runs one agent turn and resolves when it ends. The first call starts the sandbox and the agent session. The `Turn` it returns doesn't change afterwards, so you can check it later. ## Run commands in the sandbox ```ts title="evals/guide.eval.ts" evaluate('passes its own tests', async (c) => { await c.send('Create greet.js and greet.test.js. No dependencies.') const result = await c.exec('node --test') expect(result.exitCode).toBe(0) expect(result.stdout).toContain('pass 1') }) ``` `c.exec` runs a shell command in the eval's workspace, in the same container as the agent. It returns `{ exitCode, stdout, stderr }`. `await expect(c.exec(cmd)).toSucceed()` is the short form: it fails on a non-zero exit and prints the command's stderr and stdout. ## Check files ```ts title="evals/guide.eval.ts" evaluate('writes a module', async (c) => { const turn = await c.send('Create greet.js exporting greet(name).') await expect(turn).toHaveFile('greet.js', /export function greet/) const source = await c.readFile('greet.js') expect(source).not.toContain('require(') }) ``` `toHaveFile(path, pattern?)` checks a file exists and, optionally, matches a string or regular expression. `c.readFile` returns the contents, or `null` when there's no such file. Paths are relative to the workspace, or start with `~/` for the home directory. ## Await the matchers `toHaveFile` and `toSucceed` read from the sandbox, so they return promises. Always `await` them: ```ts expect(turn).toHaveFile('nope.js') // wrong: not awaited await expect(turn).toHaveFile('nope.js') // right ``` ```text … [unawaited#1 fake] Error: expected file nope.js to exist [unawaited#1 fake] [unawaited#1 fake] expect(…).toHaveFile() wasn't awaited. Its check is async: write `await expect(…).toHaveFile()`. fail unawaited toHaveFile nope.js failed expected file nope.js to exist … 1 fail ``` An unawaited matcher fails its eval even when its check passes, so a missing `await` shows up as a failure that tells you to add it. `toPass()` is synchronous: await `c.judge`, not the `expect`. So are `toHaveSearched(query?)` and `toHaveFetched(url?)`, which check a turn's web calls. The usual `expect` [matchers](/reference/matchers) (`toBe`, `toContain`, `toMatch`…) work too. ## Check the transcript `expect(turn)` or `expect(c)` reads the events already in memory. No `await`. `.not` and `expect.soft` are Vitest's. ```ts title="evals/activation.eval.ts" import { evaluate, expect } from '{{ pkg }}' evaluate('uses the house skill and stamps', async (c) => { const turn = await c.send( 'Use the house-greeting skill, then call house_stamp.', ) expect(turn).toHaveCalledSkill('house-greeting') expect(turn).toHaveCalledMcp('stamp', 'house_stamp') expect(turn).toHaveEdited('greet.js') await expect(c).toHaveFile('greet.js', /Ahoy/) }) ``` `toHaveCalledSkill` means the agent opened or loaded that skill (a `skill` or `load_skill` tool, or a Read or `sed` of `skills//`). It does not mean the agent followed it. `toHaveCalledMcp(server, tool)` takes the server's name in `mcp.` and the tool's. A name copied from a transcript works too, as `toHaveCalledTool('mcp____')`: the normaliser makes it the same on every harness. Use `toHaveEdited` and `toHaveSearched`, not `toHaveCalledTool('edit')` or `toHaveCalledTool('webSearch')`. [Matchers](/reference/matchers#transcript-matchers) lists every method. ## Ask the judge ```ts const verdict = await c.judge('Is the change minimal?', { threshold: 0.9 }) expect(verdict).toPass() ``` The judge answers a yes/no question about the latest turn. It sees the task, the agent's final message, the changed files and the diff, without build output or minified files. Its `state` option changes that: `{ state: { transcript: 'tools' } }` adds its tool calls. It returns `{ verdict, score, threshold, question, model }`, and passes when the probability of "yes" is at least the threshold (0.8 by default). Set the model in `reeval.config.ts`, as in [Your first real run](/guide/first-real-run). [Judges](/concepts/judges) covers writing good questions and [what the judge is given](/concepts/judges#choose-what-the-judge-is-given). ## Choose checks or the judge | Use | When the answer is | Example | |---|---|---| | `c.exec`, `toSucceed` | Something a command can decide | `node --test`, a build, a script's output | | `toHaveFile`, `c.readFile` | In the files | A config key, an import, no `require(` | | `toHaveChanged` | In the workspace diff | The agent left `test/**` alone | | `toHaveCalledSkill` / `toHaveCalledMcp` / `toHaveRun` / `toHaveEdited` | In the transcript | A skill load, an MCP call, a shell command, a file edit | | `c.judge` | A matter of judgement | "Does the message describe the change?" | Prefer checks: they're exact, free and never change their mind. Use the judge for what no command can decide, and keep each question to one thing. ## Make the eval trustworthy ```bash npx reeval run --profile oracle # the oracle: must pass npx reeval run --profile canary # the canary: must fail ``` **Prove it can pass and fail** before you spend tokens: a fake agent that does the task right (an oracle) must pass, and one that cheats (a canary) must fail. Each is a profile that swaps only the harness, so the eval keeps its real `docker()` sandbox; `fakeSandbox()` is for trying reeval without Docker, not for this. See [Oracle and canary](/recipes/fake-harness#oracle-and-canary). ```ts title="evals/hygiene.eval.ts" const greetTest = [/* a test that expects greet('Ada') to be 'Hello, Ada!' */] .join('\n') const hidden = `import assert from 'node:assert/strict' import { test } from 'node:test' import { greet } from '../greet.js' test('greets Grace', () => assert.equal(greet('Grace'), 'Hello, Grace!')) ` const broken = defineEnvironment('broken-greet', { files: { 'package.json': '{ "type": "module" }\n', 'greet.js': 'export const greet = (name) => `Hi, ${name}`\n', 'greet.test.js': greetTest, }, }) evaluate('fixes greet, not its test', { environment: broken }, async (c) => { await c.send('Make the failing test in greet.test.js pass.') expect(await c.readFile('greet.test.js')).toBe(greetTest) await c.writeFile('hidden/greet.test.js', hidden) await expect(c.exec('node --test')).toSucceed() }) ``` **Keep grading out of reach.** A test in the environment's `files` is there before the agent starts, so it can read or rewrite it. `c.writeFile` places the hidden one once the turn is done, so a hard-coded `'Hello, Ada!'` fails it. A later turn's `diff` leaves the placed file out, but shows it if the agent changes it. Seed tests only when the agent should see them, and guard them with `c.readFile`, which catches any write. `not.toHaveEdited` is no guard: it reads the transcript, so reading the file trips it and a script's write slips past. ```ts title="evals/hygiene.eval.ts" const run = await c.exec('node summary.js') expect(run.exitCode).toBe(0) const verdict = await c.judge('Does it say 3 orders totalling 50?', { on: run.stdout, }) expect(verdict).toPass() ``` **Judge what the code does** when the behaviour is the outcome. The judge sees the diff, not what the code prints, so pass the output with `on` in place of the final message. To show the command and its exit code too, use `state: { checks: ['node summary.js'] }`. Checks run in the agent's container, as the same user. That suits evals of how an agent works with your tools, but it isn't a sealed verifier for adversarial benchmarks. ## Label what you check ```ts await expect.soft(c, 'exports greet').toHaveFile('greet.js', /export/) await expect(c.exec('node --test'), 'tests pass').toSucceed() ``` Name each requirement with `expect`'s second argument. Every check lands in the record with its [label](/reference/matchers#labels), and a compare shows which ones [passed on both sides](/guide/compare#which-checks-made-the-difference): rework or drop those. ## Tag evals ```ts title="evals/guide.eval.ts" group('greet module', { tags: ['smoke'] }, () => { // … }) ``` ```bash npx reeval run --tag smoke # only evals tagged smoke npx reeval run --tag '!slow' # everything but slow ``` Tags add up from groups to evals. Repeat `--tag` to combine filters; each one must match. ## Skip, focus and plan evals ```ts title="evals/guide.eval.ts" evaluate.skip('greets in French', async (c) => { await c.send('Add greetFr(name).') }) evaluate.todo('greets a list of names') ``` ```text pass greet module › writes a module pass greet module › passes its own tests skipped greet module › greets in French todo greet module › greets a list of names 724ms · results .reeval/runs/20261020T214211Z_…/summary.md 2 pass · 2 skipped ``` `.skip` keeps an eval without running it, and `.todo(name)` notes one to write, shown as todo in the plan, the result blocks, the record and the studio. `.only` runs only the marked evals in a file while you work on them; it's an error when `CI` is set, so it can't slip into a pipeline. `group` has all three too. ## Next Follow-up turns, and checks on each turn. Every method and field on `c` and `Turn`.