---
title: How reeval works
description: How reeval turns an eval file into attempts, a record and the studio.
---
> ## Documentation Index
> Fetch the complete documentation index at: https://docs.reeval.ai/llms.txt
> Use this file to discover all available pages before exploring further.

reeval reads your eval files, plans every attempt they ask for, runs each one in a fresh sandbox, and writes what happened to a record you can commit and browse. These pages explain each step and why it works the way it does.

```mermaid
flowchart LR
  E["evals/*.eval.ts"] --> P[plan]
  P --> A["attempts, each in its own sandbox"]
  A --> R["record: .reeval/runs, then .history"]
  R --> S["studio, history, compare"]
```

## From eval file to studio

Every `reeval run` goes through the same five steps. `--dry-run` stops after the plan.

| Step | What reeval does | Read more |
|---|---|---|
| Eval file | Loads `group` and `evaluate` calls and resolves their options | [Options and inheritance](/concepts/options) |
| Plan | Expands each eval into cells and attempts, and prints the plan line | [The matrix](/concepts/matrix) |
| Attempts | Runs each attempt in its own sandbox: agent turns, then checks and the judge | [Attempt lifecycle](/concepts/attempt-lifecycle) |
| Record | Writes each attempt's outcome, transcript, diff and timings, and each cell's pass@1 and lift | [Gates and lift](/concepts/lift) |
| Studio | Shows runs, cells and attempts; `reeval compare` diffs two runs | [Fingerprints and drift](/concepts/fingerprints) |

## The vocabulary

The code, the CLI and these docs use the same words. Coming from Inspect, Harbor or SWE-bench? The last column maps them to the words you know.

| Term | Meaning | Elsewhere called |
|---|---|---|
| eval | One `evaluate(name, fn)`: a task for the agent and the checks on its result | task, test case (Inspect, Harbor, Anthropic) |
| case | One row of `evaluate.for(cases)` | sample, example, instance |
| harness | The agent that does the task, such as `cursor({ model: 'composer-2.5' })` | agent harness, scaffold, solver. Not an *evaluation* harness: reeval is that |
| sandbox | Where the agent runs and what's baked into its image, such as `docker()` | sandbox (Inspect), environment (Harbor), container |
| environment | What the agent gets per attempt: skills, MCP servers, rules, files, setup | configuration, condition |
| control | The `compare` baseline, else the eval's environment (an empty `baseline` when it has none) | baseline, control arm |
| variant | One environment in `compare.variants`, run on its own | treatment, arm |
| cell | One harness × environment × sandbox of an eval, e.g. `cursor:composer-2.5/-/docker` | a factorial-design cell |
| attempt | One run of a cell in a fresh sandbox, e.g. `cursor:composer-2.5/-/docker#1`; `runs: n` sets how many | trial (Anthropic, Harbor), epoch (Inspect), repetition |
| run | One `reeval run`: every attempt it started, saved as one record | eval run, job (Harbor), eval log (Inspect) |
| gate | How a cell's attempts decide its outcome: `'all'` (default) or `{ passRate }` | pass^k for `'all'`, a pass@1 threshold for `{ passRate }`; epoch reducer (Inspect), quality gate |
| pass@1 | A cell's passes over its passes and fails; `pass_at_1` in `reeval query` | pass@1, success rate, resolve rate (SWE-bench) |
| checks, judge | `expect`, `c.exec` and `c.judge` on the attempt's result | grader, verifier, scorer, evaluator |
| transcript | `transcript.jsonl`: every message and tool call of an attempt | trajectory, trace |
| lift | A variant's pass@1 minus its control's, with a 95% interval | treatment effect, delta, uplift |
| effort | Measured tool calls, agent time and tokens | efficiency, cost |
| insight | A model's sentence on an eval's result, citing its facts; never changes the verdict ([Insights](/concepts/insights)) | summary, explanation |
| remark | A deterministic note on the eval itself, such as a flaky check, with the line to change | lint, warning |

A cell passes or fails by its [gate](/concepts/lift); an eval fails when any of its gated cells fails.

## Where results go

```text
.reeval/runs/<runId>.json          every run, scratch (gitignore .reeval/)
evals/.history/<runId>.json    recorded runs, which you commit
.reeval/debug/<runId>/…            local debug traces, never recorded
```

`reeval record` (or `record: 'ci'` in CI) copies a scratch run into the history folder. `reeval history`, `reeval compare`, `reeval query` and the studio read both.

## Next

<CardGroup cols={2}>
  <Card title="The matrix" icon="layer-group" href="/concepts/matrix">
    How harnesses, variants and runs multiply into attempts.
  </Card>
  <Card title="Attempt lifecycle" icon="play" href="/concepts/attempt-lifecycle">
    What happens inside one attempt, phase by phase.
  </Card>
  <Card title="Gates and lift" icon="chart-line" href="/concepts/lift">
    How a cell passes and how sure a lift is.
  </Card>
</CardGroup>