--- title: Evals without Docker or keys description: >- Write and debug evals against a scripted fake agent, sandbox and judge, then run the same files on a real agent. --- > ## Documentation Index > Fetch the complete documentation index at: https://docs.reeval.ai/llms.txt > Use this file to discover all available pages before exploring further. **Goal:** iterate on eval files in about a second per run, with no Docker, no API keys and no model spend. Keep a second config next to your real one. It swaps in `fakeHarness()`, `fakeSandbox()` and `fakeJudge()`; the eval files stay the same. This is for writing evals and trying configs without Docker. To prove an eval's checks pass and fail, run the fake harness in the eval's real sandbox: see [Oracle and canary](#oracle-and-canary). ## The files ```ts title="reeval.config.ts" import { defineConfig } from '{{ pkg }}' import { cursor } from '{{ pkg }}/harness' export default defineConfig({ harness: cursor({ model: 'composer-2.5' }), judge: { model: 'google/gemini-3.8-flash' }, }) ``` ```ts title="reeval.fake.config.ts" import { defineConfig, defineEnvironment } from '{{ pkg }}' import { fakeHarness } from '{{ pkg }}/harness' import { fakeJudge } from '{{ pkg }}/judge' import { fakeSandbox } from '{{ pkg }}/sandbox' export default defineConfig({ harness: fakeHarness({ files: { 'package.json': '{ "type": "module" }\n', 'greet.js': "export const greet = (name = 'world') => `Hello, ${name}!`\n", 'greet.test.js': [ "import { test } from 'node:test'", "import assert from 'node:assert/strict'", "import { greet } from './greet.js'", "test('greets', () => assert.equal(greet('Ada'), 'Hello, Ada!'))", '', ].join('\n'), }, text: 'I created greet.js and greet.test.js.', script: [ [{ reply: 'I created greet.js and greet.test.js.' }], [{ exec: 'node --test' }, { reply: 'Tests pass.' }], ], }), environment: defineEnvironment('fake', { sandbox: fakeSandbox() }), judge: { model: fakeJudge({ score: 0.95 }) }, }) ``` ```json title="package.json (scripts)" { "scripts": { "eval": "reeval run", "eval:fake": "reeval run --config reeval.fake.config.ts" } } ``` The evals are the Guide's [`greet` evals](/guide/first-real-run) and this one, which checks the second scripted turn: ```ts title="evals/scripted.eval.ts" import { evaluate, expect } from '{{ pkg }}' evaluate('runs the tests after a follow-up', async (c) => { await c.send('Create greet.js.') const turn = await c.send('Now run the tests.') const shell = turn.transcript.filter( (e) => e.type === 'tool' && e.tool === 'shell', ) expect(shell).toHaveLength(1) expect(turn.text).toBe('Tests pass.') }) ``` ## Run it ```bash npx reeval run --config reeval.fake.config.ts --log-order grouped --output-logs full ``` ```text [runs#1 fake] ▶ evals/scripted.eval.ts › runs the tests after a follow-up [runs#1 fake] +0:00 sandbox ready 21ms [runs#1 fake] +0:00 turn 1 done 2ms · 0 tool calls, 3 file edits [runs#1 fake] +0:00 turn 2 done 94ms · 1 tool call, 3 file edits … pass greet module › creates a tested greet module pass greet module › adds a default name in a follow-up turn pass runs the tests after a follow-up 1s · results .reeval/runs/20261020T214611Z_…/summary.md 3 pass Set analysis.model in reeval.config.ts for answers to what each eval asks. ``` The fake agent writes `files` on every turn, after its `script` steps for that turn. Your checks (`c.exec`, `toHaveFile`, `c.judge`) run for real against what it wrote. When a coding agent runs reeval for you, agent output prints the plan, the result blocks and the tally, with no per-attempt lines. This is a project of greeting evals on the fake harness, where the skill compare's control misses on purpose and the check that tells the arms apart shows its failure: ```bash REEVAL_UI=agent npx reeval run --config reeval.fake.config.ts ``` ```text plan run 20261020T211638Z_… · 16 evals · 23 cells · 23 attempts · ~2s serial (~426ms at concurrency 4) reeval run: 20261020T211638Z_… · 4 eval files … pass house greeting › greets in the house style arm pass plain 0 of 1 with-house-greeting 1 of 1 can't tell yet check plain with-house-greeting house greeting 0 of 1 1 of 1 expected command to succeed, exit code 1 +2 more passed question Which greet.js reads more like our house style? no pairs to judge: tries that failed their checks are left out 2s · results .reeval/runs/20261020T211638Z_…/summary.md 16 pass ``` A fake run proves your eval's logic, not the agent. Passing checks here mean the eval passes when an agent writes those files; run the real config before you trust a result. ## fakeHarness options | Option | What the fake agent does | |---|---| | `files` | Writes these files on every turn | | `text` | Replies with this final message (default `Done.`) | | `withSkill` | Uses these `files` and `text` instead when the environment gives it a skill, so a `compare` shows lift | | `script` | One list of steps per `c.send` (the last repeats): `{ exec }`, `{ write }`, `{ reply }`, `{ mcp: '/' }` | | `failWith` | Throws before the first turn, as an infra error | | `layout` | `'claude-code'`, `'antigravity'`, `'opencode'`, `'goose'`, `'pi'`, `'codex'`, `'fx'` or `'grok-build'`: reads rules and skills where that agent does, in its image | | `delayMs` | Waits this long per turn, and honours timeouts and Ctrl-C | | `ignoreAbort` | Keeps waiting out `delayMs` after a timeout or Ctrl-C, like an agent that won't stop | `{ exec }` runs as the agent and shows as a `shell` tool call; `{ mcp }` calls a stdio MCP server's tool and shows as `mcp____` (see [Evaluate an MCP server](/recipes/mcp-server)). ## Oracle and canary ```ts title="evals/hygiene.eval.ts" import { defineEnvironment, evaluate, expect } from '{{ pkg }}' const greetTest = [ "import { test } from 'node:test'", "import assert from 'node:assert/strict'", "import { greet } from './greet.js'", "test('greets', () => assert.equal(greet('Ada'), 'Hello, Ada!'))", '', ].join('\n') const greetsGrace = `node -e "import('./greet.js').then((m) => process.exit(m.greet('Grace') === 'Hello, Grace!' ? 0 : 1))"` const broken = defineEnvironment('broken-greet', { files: { 'package.json': '{ "type": "module" }\n', 'greet.js': 'export const greet = (name) => `Hi, ${name}`\n', 'greet.test.js': greetTest, }, }) evaluate('fixes greet, not its test', { environment: broken }, async (c) => { await c.send('Make the failing test in greet.test.js pass.') expect(await c.readFile('greet.test.js')).toBe(greetTest) await expect(c.exec('node --test')).toSucceed() await expect(c.exec(greetsGrace)).toSucceed() }) const orders = defineEnvironment('orders', { files: { 'orders.csv': 'id,total\n1,12.50\n2,7.50\n3,30.00\n' }, }) evaluate('summarises the orders', { environment: orders }, async (c) => { await c.send( 'Write summary.js. It prints the order count and total of orders.csv.', ) const run = await c.exec('node summary.js') expect(run.exitCode).toBe(0) const verdict = await c.judge('Does it say 3 orders totalling 50?', { on: run.stdout, }) expect(verdict).toPass() }) ``` An oracle is a fake agent that does the task right; a canary is one that cheats or gets it wrong. Make each a profile in your one config that swaps the harness and judge, and nothing else: ```ts title="reeval.config.ts" import { defineConfig } from '{{ pkg }}' import { cursor, fakeHarness } from '{{ pkg }}/harness' import { fakeJudge } from '{{ pkg }}/judge' export default defineConfig({ harness: cursor({ model: 'composer-2.5' }), judge: { model: 'google/gemini-3.8-flash' }, profiles: { // Does both tasks right oracle: { defaults: { harness: fakeHarness({ files: { 'greet.js': 'export const greet = (name) => `Hello, ${name}!`\n', 'summary.js': "console.log('3 orders, total 50.00')\n", }, }), judge: { model: fakeJudge({ score: 0.95 }) }, }, }, // Cheats on the test, leaves greet.js broken and prints nothing useful canary: { defaults: { harness: fakeHarness({ files: { 'greet.test.js': "import { test } from 'node:test'\ntest('greets', () => {})\n", 'summary.js': "console.log('done')\n", }, }), judge: { model: fakeJudge({ score: 0.1 }) }, }, }, }, }) ``` The profiles set no `sandbox`, so each eval runs in the sandbox it runs in for real: its environment's `docker()` image, `install` layers, egress policy and brokered secrets. The checks then pass or fail where they will on a real agent. It needs Docker, and no keys: ```bash npx reeval image build --profile oracle npx reeval run --profile oracle # must pass npx reeval run --profile canary # must fail ``` ```text [fixes#1 fake] - Expected [fixes#1 fake] + Received [fixes#1 fake] [fixes#1 fake] import { test } from 'node:test' [fixes#1 fake] - import assert from 'node:assert/strict' [fixes#1 fake] - import { greet } from './greet.js' [fixes#1 fake] - test('greets', () => assert.equal(greet('Ada'), 'Hello, Ada!')) [fixes#1 fake] + test('greets', () => {}) … fail fixes greet, not its test toBe import { test } from 'node:test' import… failed expected 'import { test } from \'node:test\'\nt…' to be 'import { test } from \'node:test\'\ni…' // Object.is equality fail summarises the orders toPass Does it say 3 orders totalling 50? failed judge score 0.10 is below 0.8 for "Does it say 3 orders totalling 50?" (fake) +1 more passed 3s · results .reeval/runs/20261020T214629Z_…/summary.md 2 fail ``` `--profile oracle` passes and exits 0; the canary exits 1. The rewritten test passes `node --test`, so only the `c.readFile` guard catches it. A canary that writes `greet = () => 'Hello, Ada!'` passes the seeded test too, and fails `greetsGrace`, the check the agent never sees. Give each check you rely on a canary that only it catches, and rerun them after each edit to the eval. | To check | Do | |---|---| | The real fix, not canned files | A `script` step runs in the container as the agent: `{ exec: 'node scripts/solve.mjs' }` | | That a judge question passes good work | Keep the real `judge.model` on the oracle, with its key | | Rules and skills where your agent reads them | `fakeHarness({ layout: 'claude-code', … })`, which also uses that agent's image | Don't validate checks on `fakeSandbox()`. It runs commands on your host with your tools, brokers no secrets and enforces no egress, so a check can pass there and fail in the eval's `docker()`. An eval or group that sets its own `harness` keeps it under the profile, and an ungated cell (a `gate: 'none'` study, or a control) can fail its checks without changing the exit code: read its arm's count there. ## Simulate an infra failure ```ts title="reeval.fail.config.ts" import { defineConfig, defineEnvironment } from '{{ pkg }}' import { fakeHarness } from '{{ pkg }}/harness' import { fakeSandbox } from '{{ pkg }}/sandbox' export default defineConfig({ include: ['evals/scripted.eval.ts'], harness: fakeHarness({ failWith: 'the agent CLI crashed on start' }), environment: defineEnvironment('fake', { sandbox: fakeSandbox() }), }) ``` ```bash npx reeval run --config reeval.fail.config.ts ``` ```text [runs#1 fake] ▶ evals/scripted.eval.ts › runs the tests after a follow-up [runs#1 fake] +0:00 sandbox starting [runs#1 fake] +0:00 cleaning up [runs#1 fake] infra error [runs#1 fake] the agent CLI crashed on start infra error runs the tests after a follow-up infra error the agent CLI crashed on start 765ms · results .reeval/runs/20261020T214612Z_…/summary.md 1 infra error ``` The run exits 3, so you can check how your CI treats an infrastructure error before a real one happens. ## Check where rules land `layout` makes the fake agent read rules and skills where a real one would. With `layout: 'claude-code'`, rules go to `CLAUDE.md`: ```ts title="evals/rules.eval.ts" import { defineEnvironment, evaluate, expect } from '{{ pkg }}' const houseRules = defineEnvironment('house-rules', { rules: 'Greet with "Ahoy".\n', }) evaluate( 'puts rules where Claude Code reads them', { environment: houseRules }, async (c) => { expect(await c.readFile('CLAUDE.md')).toContain('Ahoy') }, ) ``` No turn runs, so the check sees only what the environment set up: `c.readFile` starts the sandbox. ## The fake sandbox and judge | Fake | What it does | |---|---| | `fakeSandbox()` | Gives each attempt a temporary directory on the host (`dir` picks the parent). Commands run on your machine with your tools | | `fakeSandbox()` and `egress` | It can't enforce a policy: an eval with a restricted one stops the run with exit 2 (see [Offline evals](/recipes/offline)). Brokered secrets become placeholders that nothing substitutes | | `fakeJudge({ score })` | Returns that score for every question (default 1), so `fakeJudge({ score: 0.3 })` tests how a failing verdict reads | Both configs plan the same evals, so `--dry-run` under either shows the same eval rows. ## Variations | Variation | How | |---|---| | A flaky agent | `fakeHarness({ files, flakeRate: 0.3 })` writes nothing in about 30% of attempts, for trying `gate: { passRate }` | | Canned tool calls | `tools: [{ tool: 'read', input: {…}, output: '…' }]` adds tool events to every turn, for transcript checks | | Only some files | `include: ['evals/hello.eval.ts']` in the fake config leaves out evals that need a real agent or Docker | | Profiles | The fake config needs its own `profiles` if you pass `--profile`; give it the same profile names as the real config | | Record fake runs | `reeval run --config reeval.fake.config.ts --record` writes history you can open in `reeval studio`, handy for trying the studio or `reeval query` | ## Next Every option and script step. Run the real config on every pull request.