--- title: reeval evaluating reeval description: >- A real suite that asked whether a skill helps a coding agent use reeval, and what its numbers did and didn't show. --- > ## Documentation Index > Fetch the complete documentation index at: https://docs.reeval.ai/llms.txt > Use this file to discover all available pages before exploring further. **Question:** can a coding agent use reeval itself, and does a skill about reeval help? No model has seen reeval before, so a skill about it is a real variable. These runs used reeval from before PR #8, #10 and #12. Since then, Cursor's per-session files no longer make every run "not comparable", the studio no longer marks a run with only an ungated failure as Failed, and reeval computes effort itself. The numbers below are unaffected. ## The scenario | | | |---|---| | Harness | `cursor({ model: 'composer-2.5' })` in Docker, concurrency 4 | | Judge | `google/gemini-3.8-flash`, threshold 0.8 | | Cells per eval | control `broken-project`, plus `reeval-weak` and `reeval-good`, via `compare: { baseline: brokenProject, variants: [reevalWeak, reevalGood] }` | | Runs | `runs: 2`, `gate: { passRate: 0.5 }`, budget 20 attempts and 1 hour | | Sandbox | reeval's image plus npm 11 and the packed reeval tarball, from a `FROM ${REEVAL_BASE}` Dockerfile | | Egress | `DEFAULT_EGRESS_ALLOW`: npm, PyPI and GitHub, plus Cursor's own hosts | Inside the sandbox the agent has no Docker and no API key, so it has to use reeval's fake harness, sandbox and judge. The checks run reeval in the agent's project and read the plan as JSON. `reeval-good` is about 230 lines covering the whole API with a table of common mistakes. `reeval-weak` is about 20 lines, and wrong: it names the wrong package, config file and options. ## The files Two skills and a broken project, as environments: ```ts title="evals/envs.ts" import { defineEnvironment } from '{{ pkg }}' const broken = '../fixtures/broken-project' /** A reeval project someone got wrong. The compare baseline. */ export const brokenProject = defineEnvironment( 'broken-project', ({ file }) => ({ files: { 'package.json': file(`${broken}/package.json`), 'reeval.config.ts': file(`${broken}/reeval.config.ts`), 'evals/environments.ts': file(`${broken}/evals/environments.ts`), 'evals/docs.eval.ts': file(`${broken}/evals/docs.eval.ts`), 'docs/api.md': file(`${broken}/docs/api.md`), }, setup: [{ run: 'npm install --no-audit --no-fund', timeout: '3m' }], }), ) /** A hasty skill on that project: shallow, vague and partly out of date. */ export const reevalWeak = brokenProject.extendEnvironment( 'reeval-weak', ({ skill }) => ({ skills: [skill.local('../skills/reeval-weak')], }), ) /** An in-depth, accurate skill for writing and running reeval evals. */ export const reevalGood = brokenProject.extendEnvironment( 'reeval-good', ({ skill }) => ({ skills: [skill.local('../skills/reeval-good')], }), ) ``` The hardest of the three evals. `dryRun` runs `npx --no-install reeval run --dry-run --json` in the sandbox and parses the plan: ```ts title="evals/fix-config.eval.ts" import { evaluate, expect, group } from '{{ pkg }}' import { dryRun } from './checks.ts' import { brokenProject, reevalGood, reevalWeak } from './envs.ts' const task = 'This is a reeval project (reeval is an eval runner for coding agents; npm ' + 'package {{ pkg }}, CLI `reeval`, already installed). `npx reeval run ' + '--dry-run` fails. Fix reeval.config.ts and the evals so the plan is what ' + 'the author meant: the docs eval compares no environment against the ' + 'with-docs environment, 3 runs per cell, on the fake harness. Then list ' + 'each mistake you fixed, one line each.' const question = 'Pass only if the final message names all three mistakes correctly: ' + 'compare cannot be set in reeval.config.ts (it goes on the group or eval); ' + 'environment takes one environment, not a list; compare takes ' + '{ baseline, variants } and does not layer them; ' + 'and the option is runs, not repeat.' group( 'broken project', { compare: { baseline: brokenProject, variants: [reevalWeak, reevalGood] } }, () => { evaluate('fixes a broken config and explains it', async (c) => { await c.send(task) const verdict = await c.judge(question) const { exitCode, plan } = await dryRun(c) expect(exitCode, 'reeval run --dry-run --json exits 0').toBe(0) expect(plan?.totals).toMatchObject({ evals: 1, cells: 2, attempts: 6 }) await expect(c.exec('npx --no-install reeval run --ui plain')).toSucceed() expect(verdict).toPass() }) }, ) ``` reeval's own errors point at the mistakes one at a time. Run in the broken project, before and after the first fix: ```text reeval: compare can't be set in reeval.config.ts: it would multiply every eval. Put it on the group or eval that asks the question. ``` ```text reeval: evals/docs.eval.ts: Unknown option "repeat" in evaluate("answers from the docs"). Did you mean "runs"? ``` ## The plan ```text plan 3 evals · 9 cells · 18 attempts · ~54m 00s serial (~13m 30s at concurrency 4) · budget 20 attempts ✓, 1h eval harnesses variants runs attempts notes evals/compare.eval.ts › from scratch › plans a skill compare 1 2 2 6 egress: allow registry.npmjs.org, … evals/fix-config.eval.ts › broken project › fixes a broken… 1 2 2 6 egress: allow registry.npmjs.org, … evals/scaffold.eval.ts › from scratch › scaffolds a first eval… 1 2 2 6 egress: allow registry.npmjs.org, … ``` The run took about 4 minutes, so the first estimate was far too high. It improves once there's history to estimate from. ## Pass rates: lift +0 Two recorded runs, the second with `CI=1`. Its summary, in the format the reeval of the time printed (`reeval run` now prints a [result block](/concepts/output#result-blocks) per eval): ```text evals/compare.eval.ts from scratch ✓ plans a skill compare 6/6 · pass@1 100% gated 6 attempts cursor:composer-2.5/-/reeval-tarball control 2/2 pass@1 100% cursor:composer-2.5/reeval-weak/reeval-tarball variant 2/2 pass@1 100% lift +0 pts (95% −66…+66, 2/2 vs 2/2), within noise gate passRate 0.5 cursor:composer-2.5/reeval-good/reeval-tarball variant 2/2 pass@1 100% lift +0 pts (95% −66…+66, 2/2 vs 2/2), within noise gate passRate 0.5 evals/fix-config.eval.ts broken project ✓ fixes a broken config and explains it 5/6 · pass@1 100% gated 6 attempts cursor:composer-2.5/broken-project/reeval-tarball control 1/2 pass@1 50% not gated cursor:composer-2.5/reeval-weak/reeval-tarball variant 2/2 pass@1 100% lift +50 pts (95% −27…+91, 2/2 vs 1/2), within noise gate passRate 0.5 cursor:composer-2.5/reeval-good/reeval-tarball variant 2/2 pass@1 100% lift +50 pts (95% −27…+91, 2/2 vs 1/2), within noise gate passRate 0.5 … Attempts 17 passed · 1 missed, not gated (18 attempts, 3 evals) Duration 3m 54s ``` Over both runs, 35 of 36 attempts passed: control 11/12, `reeval-weak` 12/12, `reeval-good` 12/12. Lift was +0 on two evals in both runs; the +50 on the third is one control failure, and its interval crosses zero. ## What actually differed: effort The skill's real effect was in effort, not in the pass rate. At the time it took a script over the records' `activity` and `timings`: | Eval | Cell | Median agent time | Median tool calls | |---|---|---|---| | scaffolds a first eval | control | 45 s | 42 | | | `reeval-good` | 21 s | 4 | | plans a skill compare | control | 56 s | 50 | | | `reeval-good` | 27 s | 10 | | fixes a broken config | control | 31 s | 28 | | | `reeval-good` | 30 s | 14 | Overall, `reeval-good` needed 4× fewer tool calls (median 10 against 40) and about 35% less agent time (26 s against 41 s). Every control transcript read the tarball's compiled `dist/`; the package shipped no README. `reeval-weak` matched the control: no attempt followed its wrong advice, because the prompt's concrete facts overrode it. ## What the judge caught | Run | Score | What happened | |---|---|---| | The full run | 35 of 36 at 0.98 or higher | The judge passed the work | | The full run, control on `fix-config` | 0.10 | It fixed the config, but its explanation said `baseline` "added nothing" and `repeat` "is only an alias". Neither is true: `baseline` doesn't exist and `repeat` isn't an alias | | The smoke run, a correct scaffold | 0.05 | The question named specific matchers and the agent used others: a bug in the question | The judge was right about `fix-config`, and that was the one quality difference the run found. The verdict carried no reason, so finding the false claims meant reading the final message. ## Lessons | Lesson | Why | |---|---| | Tasks the agent already solves saturate | composer-2.5 solved all three tasks from the type definitions and reeval's error messages. To separate variants, make the task harder or measure effort | | Lift measures pass rate only | The skill's 4× effect appeared nowhere in `lift`. The `effort:` line under each variant, `reeval query effort` and [`reeval analyze`](/guide/analyze) report it | | Name outcomes, not methods, in judge questions | "Checks hello.txt with a deterministic assertion" survives; "uses `toHaveFile`" fails correct work | | Two runs a cell is a smoke test | Every interval here is more than 100 points wide | The same suite later ran on Cursor and Antigravity side by side; see [Compare agents](/recipes/compare-agents). ## Next The skill compare, as a recipe. What the interval means, and when to trust it.