---
title: The run record
description: Every field of a run's JSON record, and the history folder layout.
---
> ## Documentation Index
> Fetch the complete documentation index at: https://docs.reeval.ai/llms.txt
> Use this file to discover all available pages before exploring further.

Each run is one JSON file, `<runId>.json`, plus optional artifacts in `<runId>/`. The record is immutable once written. `reeval show <run> --json` prints it.

A record written before a field existed still loads everywhere: reeval reads the missing field as empty (`[]`, or `false` for `todo`). A record from a newer reeval loads too, ignoring fields this reeval doesn't know, unless its `schemaVersion` is higher: then reeval skips it with a warning to upgrade.

```bash
npx reeval show latest --json | jq '.evals[] | select(.outcome != "pass") | .id'
```

## Layout

```text
evals/.history/                            commit this
  .gitignore                               un-ignores recorded files (`*.log` and the rest)
  <runId>.json                             the record
  <runId>/summary.md                       the run as Markdown
  <runId>/insights.json                    insights and remarks (see Insights)
  <runId>/worktree.patch                   uncommitted changes, when the run was dirty
  <runId>/<eval-slug>/questions/<n>.json   what pairwise question n's model was sent
  <runId>/<eval-slug>/<cell-slug>/<n>/     one directory per attempt
    transcript.jsonl                       timed agent events (see Transcripts)
    diff.patch                             the agent's changes, dependency noise dropped
    judge-input.json                       what each judge call saw
    setup.log                              each environment setup step's output
    checks/<n>.log                         each c.exec: command, exit code, stdout, stderr
    artifacts/<name>.png                   each c.capture() PNG (`history.artifacts` keeps it)
    questions/<n>.json                     what question n's model was sent about this attempt
.reeval/                                       scratch: gitignore it
  current                                  the run id minted (or resumed) this session
  staging/<runId>/                         a run in progress, until assemble
  runs/<runId>.json + <runId>/             every reeval run, same shape
  debug/<runId>/…                          local debug data, never recorded
  history.db                               reeval query's projection
```

The record is `kind: "reeval-run"`, `schemaVersion: 1`. The schema only grows: a newer reeval reads an older record, and an older reeval skips a record with a higher `schemaVersion`, saying to upgrade.

Single tool outputs, check streams and per-file diffs over 128 KB keep their head and tail, a truncation marker and the full text's sha256. Records are redacted before they're written.

## Reading one

Start at `evals[]` where `outcome != "pass"`, then its `cells[]` for pass@1 (`passRate`), roles and lift, then each failing attempt's `error` and `judgements`. Open the files its `files` points to.

```json title="evals/.history/<runId>.json (trimmed)"
{
  "kind": "reeval-run",
  "schemaVersion": 1,
  "runId": "20260926T170558Z_bd6205e_ada-lovelace_sycz",
  "status": "complete",
  "summary": {
    "evals": 1, "attempts": 4, "pass": 2, "fail": 2,
    "notGated": 2, "infraError": 0, "skipped": 0
  },
  "evals": [{
    "id": "evals/skill.eval.ts › house greeting › greets in the house style",
    "outcome": "pass",
    "…": "…"
  }]
}
```

## Fields

The type definitions are in `packages/reeval/src/history/types.ts`; the environment, egress and image records are in `src/types.ts`. This section is generated from them.

### Value types

[//]: # (generated:record-values)

| Type | Values |
|---|---|
| `Outcome` | `'pass'`, `'fail'`, `'infra_error'`, `'skipped'` |
| `ErrorKind` | `'assertion'`, `'timeout'`, `'aborted'`, `'infra'`, `'judge'` |
| `TurnStatus` | `'completed'`, `'aborted'`, `'error'` |
| `ArtifactLevel` | `'all'`, `'failures'`, `'none'` |
| `RecordMode` | `'manual'`, `'ci'`, `'always'` |
| `FamilyVerdict` | `'lift'`, `'drop'`, `'noise'`, `'insufficient'` |

[//]: # (/generated:record-values)

`runner` is `{ kind: 'ci', provider, runUrl, attempt, pr: { number, baseRef } | null }` in CI, or `{ kind: 'local', hostHash }` on your machine.

An attempt's `error` is `{ kind, message, prepare?, resource? }`:

| Field | Set when | Shape |
|---|---|---|
| `prepare` | Its sandbox couldn't be prepared before the run (see [Sandbox providers](/reference/sandbox/providers)), the same on every attempt that needed it | `{ class, stage, ref, cause, log, setBy }`, `class` one of `config`, `transient` or `resource` |
| `prepare` | A git `workspace` couldn't be fetched | `stage: "workspace"`, the workspace as `ref`, such as `../repo#v9` |
| `resource` | The kernel killed `kills` processes at the sandbox's memory limit | `{ cause: 'oom', kills }` |

An attempt with `prepare` never started, so it has no artifacts directory. One with `resource` has `kind: infra` and outcome `infra_error`, whatever its checks said, so it's left out of pass rates and lift.

[//]: # (generated:record-types)

### RunRecord

<Property name="kind" type="'reeval-run'"></Property>

<Property name="schemaVersion" type="typeof SCHEMA_VERSION"></Property>

<Property name="runId" type="string">
  Also the file name: `<UTC start>_<short sha>_<author or ci>`.
</Property>

<Property name="status" type="'complete' | 'interrupted' | 'budget'">
  `budget` when `budget.duration` stopped new attempts from starting.
</Property>

<Property name="startedAt" type="string"></Property>

<Property name="endedAt" type="string"></Property>

<Property name="durationMs" type="number"></Property>

<Property name="author" type="string"></Property>

<Property name="git" type="GitInfo | null"></Property>

<Property name="runner" type="RunnerInfo"></Property>

<Property name="dependencies" type="{lockfiles: Array<{path: string; sha256: string}>}"></Property>

<Property name="stack" type="Stack"></Property>

<Property name="environments" type="EnvironmentRecord[]">
  Each environment the run applied, fingerprinted; secret values are never included.
</Property>

<Property name="profile" type="{name: string; reason: string; defaults: {harness?: string[]; environment?: string; sandbox?: string; runs?: number; timeout?: string | number; env?: string[]}; limit: {runs?: number; compare?: false | {maxVariants: number}; harness?: string[]}} | null">
  The profile the run was sized with, and why it was chosen.
</Property>

<Property name="plan" type="{evals: number; cells: number; attempts: number; estimateMs: number; sandboxWarnings: SandboxWarning[]}">
  What the run planned before it started, and each level whose sandbox ran as root or with capabilities added back.
</Property>

<Property name="budget" type="{attempts?: number; duration?: string | number} | null"></Property>

<Property name="config" type="{sandbox: string; environment: string | null; runs: number; judgeModel: string | null; threshold: number; timeoutMs: number; concurrency: number; configHash: string}">
  The run's settings. `sandbox` is the id of each sandbox the planned attempts ran on, after the profile's defaults, joined with `, `.
</Property>

<Property name="cost" type="Cost"></Property>

<Property name="artifacts" type="ArtifactLevel">
  Which artifacts sit next to this record. Refs to files that weren't kept are removed.
</Property>

<Property name="redaction" type="{version: 1}"></Property>

<Property name="summary" type="RunSummary"></Property>

<Property name="evals" type="EvalEntry[]"></Property>

<Property name="families" type="FamilyRecord[]" optional>
  Each pooled `evaluate.each` that ran; unset when none did.
</Property>

<Property name="errors" type="string[]" optional>
  Errors outside any attempt, redacted: an eval file that didn't load, a rejection or throw no attempt awaited, a worker that died. Any one fails the run. Unset when there were none.
</Property>

### GitInfo

<Property name="sha" type="string">
  The commit the run executed at: HEAD when `reeval run` started. The later commit that adds the history file is a different, descendant commit.
</Property>

<Property name="branch" type="string | null"></Property>

<Property name="dirty" type="boolean">
  Uncommitted changes, outside `.reeval/` and the history folder, when the run started.
</Property>

<Property name="diffHash" type="string | null">
  sha256 of the uncommitted changes (tracked and untracked, as a binary patch), when dirty.
</Property>

<Property name="worktreePatch" type="FileRef | null">
  Those changes as `worktree.patch` in the artifact dir; apply on `sha` to reproduce.
</Property>

<Property name="worktreePatchNote" type="string | null">
  Why a dirty run has no `worktree.patch`: too large, artifacts: 'none', or it looked like it held a secret (then where, and which rule, never the value).
</Property>

<Property name="commitTime" type="string"></Property>

<Property name="shallow" type="boolean"></Property>

<Property name="remote" type="string | null">
  The `origin` URL with any credentials removed.
</Property>

### Stack

What ran and graded the agents: the machine, images, harnesses and judge.

<Property name="reeval" type="string">
  The reeval version that wrote the run.
</Property>

<Property name="node" type="string">
  `process.versions.node`: under Bun or Deno, the Node version they're compatible with.
</Property>

<Property name="runtime" type="{name: string; version: string}" optional>
  The runtime reeval ran on, `node`, `bun`, `deno` or another, and its own version. Absent in runs recorded before it, which ran on Node.
</Property>

<Property name="language" type="{name: string; version?: string}" optional>
  The language the evals are written in, `typescript` for `*.eval.ts`, and the version of its toolchain when the runner reports one. Absent in runs recorded before it.
</Property>

<Property name="os" type="string"></Property>

<Property name="arch" type="string"></Property>

<Property name="images" type="Array<AttemptImage &amp; {repoDigests: string[]}>">
  Every image an attempt started from. `id` is content-addressed; `repoDigests` only exist once pushed.
</Property>

<Property name="packages" type="Record<string, string>">
  Exact versions of the packages that run and grade the agent.
</Property>

<Property name="harnesses" type="Array<{id: string; kind: string; model: string | null; effort: string | null; resolvedModels: string[]; resolvedEffort: string[]}>">
  Each harness: its configured `model` and requested `effort`, `resolvedModels` (what it reported serving, with the variant it resolved), and `resolvedEffort` (reasoning-effort suffixes those labels named). Fast (`[fast=true]`) is not `resolvedEffort`.
</Property>

<Property name="judge" type="{model: string | null; resolvedModels: string[]; threshold: number; version: string; state?: ResolvedJudgeStateOptions}">
  The run's judge: the config's, with what the profile's `judge` and `defaults.judge` set over it. `config.judgeModel` and `config.threshold` name the same one.
</Property>

<Property name="context" type="ContextFile[]">
  Union of every attempt's context files.
</Property>

<Property name="fingerprintScope" type="string[]">
  What eval fingerprints hash; fingerprints from different scopes aren't comparable.
</Property>

### RunSummary

Counts attempts, except `evals`.

<Property name="evals" type="number"></Property>

<Property name="attempts" type="number"></Property>

<Property name="pass" type="number"></Property>

<Property name="fail" type="number">
  Every failed attempt, including `notGated` ones.
</Property>

<Property name="notGated" type="number">
  Failed attempts in ungated cells (controls, by default), which don't fail the run.
</Property>

<Property name="infraError" type="number"></Property>

<Property name="skipped" type="number"></Property>

### SandboxWarning

A sandbox with more privilege than the default `reeval` user: `docker({ root: true })`, or `docker({ capabilities })`. One per level that set such a sandbox, however many evals inherit it.

<Property name="kind" type="'root' | 'capabilities'"></Property>

<Property name="level" type="'config' | 'profile' | 'group' | 'eval'">
  Which level set the sandbox.
</Property>

<Property name="where" type="string">
  That level as reeval names it: `reeval.config.ts`, `profiles.pr.defaults`, `group("net")` or `evaluate("installs deps")`.
</Property>

<Property name="file" type="string | null">
  The eval file that sets it, for a group or eval; null for the config or a profile.
</Property>

<Property name="capabilities" type="string[]" optional>
  The capabilities added back, for `capabilities`.
</Property>

<Property name="message" type="string">
  The line reeval prints, after `WARNING: ` (root) or `warning: ` (capabilities).
</Property>

### Cost

The tokens the run used, as the harnesses count them; reeval prices nothing.

<Property name="inputTokens" type="number | null">
  Prompt tokens, cache reads and writes included.
</Property>

<Property name="outputTokens" type="number | null"></Property>

<Property name="usage" type="TokenUsage | null">
  The same totals split as each turn reports them.
</Property>

<Property name="source" type="'reported' | 'unreported'"></Property>

### EvalEntry

<Property name="id" type="string">
  `file › group › … › name`, as shown.
</Property>

<Property name="key" type="string">
  What the eval is matched on across runs; currently the same as `id`.
</Property>

<Property name="slug" type="string">
  Directory name inside the artifact directory, from `key`.
</Property>

<Property name="file" type="string"></Property>

<Property name="group" type="string[]"></Property>

<Property name="name" type="string"></Property>

<Property name="fingerprint" type="string">
  Results are comparable only when fingerprints match.
</Property>

<Property name="sources" type="Array<{path: string; sha256: string}>">
  The eval file and every local file it imports, directly or not, by content hash.
</Property>

<Property name="tags" type="string[]"></Property>

<Property name="compare" type="EvalCompareMeta" optional>
  The baseline environment, when the eval uses `compare`.
</Property>

<Property name="outcome" type="Outcome">
  Over gated cells: `fail` if any failed, else `infra_error` if any is (a cell is `infra_error` only when nothing counted), else `skipped` if all were, else `pass`. An ungated cell's failures don't fail the eval.
</Property>

<Property name="todo" type="boolean">
  A placeholder, `evaluate.todo` or `group.todo`: it never runs, and its outcome is `skipped`.
</Property>

<Property name="passRate" type="number | null">
  pass@1, pass / (pass + fail), over the gated cells' attempts; infra errors and skips excluded.
</Property>

<Property name="cells" type="CellRecord[]"></Property>

<Property name="attempts" type="AttemptRecord[]"></Property>

<Property name="answers" type="CompareAnswer[]" optional>
  The eval's pairwise questions (those with `across`), answered after its attempts by votes, one per question; unset without any. They annotate: `outcome` and `passRate` never depend on them. Each vote may include a short `reason` (capped). `reeval analyze` includes each answer in the digest (rates by cut, votes, ties, a failed call, a sample of reasons). Each answer's `input` is `<eval-slug>/questions/<n>.json`: every call its model was sent, per pair and order, secrets masked and screenshots as refs; unset where the eval's files weren't kept.
</Property>

<Property name="family" type="{id: string; name: string; case: number; cases: number}" optional>
  Set on each case of a pooled `evaluate.each`; `RunRecord.families` has the suite.
</Property>

### CellRecord

One harness and environment of an eval, and how its attempts went.

Also has every field of `Cell`.

<Property name="fingerprint" type="string">
  `sha256` of `fingerprintParts`; results are comparable only when it matches.
</Property>

<Property name="fingerprintParts" type="FingerprintParts"></Property>

<Property name="environmentDigest" type="string | null">
  The environment's digest: the join key into `RunRecord.environments`; null for none.
</Property>

<Property name="harnessKind" type="string"></Property>

<Property name="gateway" type="string" optional>
  The gateway in the config's `gateway` list its harness resolved to (`direct` is each provider's own key); unset when no config, profile or pin named one.
</Property>

<Property name="model" type="string | null">
  The configured harness model, when it has one.
</Property>

<Property name="effort" type="string | null">
  The reasoning effort the harness factory asked for; null when it left it unset. Fast is not this.
</Property>

<Property name="gate" type="CellGate">
  How the attempts decide the cell's outcome; `'none'` when it doesn't fail the eval.
</Property>

<Property name="overrides" type="string[]">
  Environment keys the variant replaced on the control's (or a child on its parent's).
</Property>

<Property name="runs" type="number"></Property>

<Property name="pass" type="number"></Property>

<Property name="fail" type="number"></Property>

<Property name="infraError" type="number"></Property>

<Property name="skipped" type="number"></Property>

<Property name="passRate" type="number | null">
  pass@1: pass / (pass + fail); infra errors and skips excluded.
</Property>

<Property name="outcome" type="Outcome">
  By `gate`. `'all'` (pass^k: every counted attempt passes) matches `{ passRate: 1 }`; `{ passRate }` is a pass@1 threshold. Infra only when nothing counted; `'none'` reports the worst outcome. A variant below its `minLift` fails too.
</Property>

<Property name="lift" type="Lift" optional></Property>

<Property name="checks" type="CellCheck[]" optional>
  Each check (assertion label) seen in any attempt of the eval, over this cell's counted attempts; on a variant cell, the control's beside it. Unset when no attempt recorded any.
</Property>

### Lift

A variant cell's pass@1 against the control's, on the same harness and sandbox.

<Property name="vs" type="string">
  The control cell's key.
</Property>

<Property name="control" type="{pass: number; counted: number}">
  Passes out of counted attempts (passes and fails) on each side.
</Property>

<Property name="variant" type="{pass: number; counted: number}"></Property>

<Property name="points" type="number">
  Variant pass@1 minus control pass@1, from -1 to 1.
</Property>

<Property name="interval" type="{lo: number; hi: number; level: 0.95; method: 'newcombe-wilson'}">
  95% interval on the difference, by Newcombe's hybrid score method from two Wilson intervals.
</Property>

<Property name="withinNoise" type="boolean">
  The interval spans 0: the attempts can't tell the variant from the control.
</Property>

<Property name="minLift" type="number" optional>
  Set when the compare has a `minLift`; the cell fails when `interval.lo` is below it.
</Property>

### FamilyRecord

A pooled `evaluate.each`: its cases as samples of one suite. Each case is also an eval.

<Property name="id" type="string">
  `file › group › … › template`.
</Property>

<Property name="file" type="string"></Property>

<Property name="group" type="string[]"></Property>

<Property name="name" type="string">
  The name template, e.g. `fixes $id`.
</Property>

<Property name="cases" type="number">
  Cases in the list.
</Property>

<Property name="ran" type="number">
  Cases with an eval in this run; fewer when a filter or tag left some out.
</Property>

<Property name="rates" type="FamilyRate[]">
  One per cell.
</Property>

<Property name="lifts" type="FamilyLift[]">
  One per variant cell, with `compare`.
</Property>

<Property name="outcome" type="'pass' | 'fail'">
  `fail` when a lift is below its `minLift`. The cases still pass or fail on their own.
</Property>

### FamilyRate

One cell's pass@1 over a family's cases.

<Property name="cell" type="string">
  The cell's key, the same in every case.
</Property>

<Property name="cases" type="number">
  Cases with at least one counted attempt in this cell.
</Property>

<Property name="pass" type="number">
  Passes and counted attempts over those cases.
</Property>

<Property name="counted" type="number"></Property>

<Property name="rate" type="number">
  The mean of the cases' pass@1; at one attempt a case, `pass / counted`.
</Property>

<Property name="interval" type="FamilyInterval"></Property>

### FamilyLift

A variant cell against its control over a family's cases, paired by case.

<Property name="variant" type="string">
  The variant cell's key.
</Property>

<Property name="vs" type="string">
  The control cell's key.
</Property>

<Property name="pairs" type="number">
  Cases with counted attempts on both sides.
</Property>

<Property name="variantRate" type="number">
  Mean pass@1 over the paired cases on each side.
</Property>

<Property name="controlRate" type="number"></Property>

<Property name="discordant" type="{variantOnly: number; controlOnly: number}" optional>
  Set at one attempt a case and side: pairs where only one side passed.
</Property>

<Property name="points" type="number">
  Variant minus control, from -1 to 1.
</Property>

<Property name="interval" type="FamilyInterval"></Property>

<Property name="p" type="number | null">
  McNemar's exact p (one attempt a case), else Wilcoxon signed-rank over the cases.
</Property>

<Property name="withinNoise" type="boolean">
  The interval spans 0.
</Property>

<Property name="verdict" type="FamilyVerdict"></Property>

<Property name="minLift" type="number" optional>
  The compare's `minLift`: on a family it gates this lower bound, not each case's.
</Property>

<Property name="outcome" type="'pass' | 'fail'">
  `fail` when `interval.lo` is below `minLift`.
</Property>

<Property name="effort" type="FamilyEffort" optional></Property>

### FamilyInterval

A 95% interval on a pooled rate or difference.

<Property name="lo" type="number"></Property>

<Property name="hi" type="number"></Property>

<Property name="level" type="0.95"></Property>

<Property name="method" type="'wilson' | 'newcombe-paired' | 't-cases'">
  `wilson` / `newcombe-paired`: one counted attempt a case and side, so each case is one Bernoulli trial (or pair). `t-cases`: several, so a t interval over the cases' pass@1, with attempts within a case never counted as independent.
</Property>

### FamilyEffort

The variant's effort against the control's, paired by case.

<Property name="tools" type="FamilyEffortMetric"></Property>

<Property name="agentMs" type="FamilyEffortMetric"></Property>

<Property name="tokens" type="FamilyEffortMetric" optional>
  Over the cases where every counted attempt reported usage; omitted when none did.
</Property>

### FamilyEffortMetric

One effort metric over a family's paired cases.

<Property name="pairs" type="number">
  Cases with a value on both sides.
</Property>

<Property name="ratio" type="number | null">
  The median over cases of the variant's per-case median, over the control's; null at 0.
</Property>

<Property name="p" type="number | null">
  Wilcoxon signed-rank over the per-case differences; null when every difference is 0.
</Property>

<Property name="verdict" type="'fewer' | 'more' | 'noise' | 'insufficient'">
  `insufficient` below 6 non-zero differences, where the exact test can't reach 0.05. `fewer` and `more` are the test's direction (which side's differences rank higher), not the median's.
</Property>

### FingerprintParts

Each input of a cell fingerprint, hashed on its own, so a change can be named.

<Property name="sources" type="string">
  Over `EvalEntry.sources`.
</Property>

<Property name="harness" type="string">
  Over the harness descriptor, less concurrency caps.
</Property>

<Property name="environment" type="string | null"></Property>

<Property name="image" type="string | null"></Property>

<Property name="agentCli" type="string | null"></Property>

<Property name="judge" type="string">
  Over the judge's model, threshold and version, and the config's `state` when it sets one.
</Property>

<Property name="context" type="string">
  Over the attempt's context files.
</Property>

<Property name="gateway" type="string" optional>
  The gateway its model calls went through; unset on `direct`, each provider's own key.
</Property>

### AttemptRecord

<Property name="key" type="string">
  `<eval key> # <cell key>#<n>`: unique in the run.
</Property>

<Property name="id" type="string">
  `<cell key>#<n>`: unique in its eval.
</Property>

<Property name="cell" type="string">
  The cell's key; see `EvalEntry.cells`.
</Property>

<Property name="n" type="number">
  Run number within the cell, from 1.
</Property>

<Property name="seq" type="number | null">
  Start order across the run, from 1; null for attempts that never started.
</Property>

<Property name="startedAt" type="string | null"></Property>

<Property name="imageId" type="string | null">
  The image the container started from, in `stack.images`; null for the fake sandbox.
</Property>

<Property name="env" type="{vars: string[]; secrets: string[]; brokered: string[]}">
  Names of the variables and secrets set in the sandbox, and the brokered ones; never values.
</Property>

<Property name="context" type="ContextFile[]">
  Skills, rules and MCP config visible to the agent in its sandbox, by content hash.
</Property>

<Property name="egress" type="EgressRecord | null">
  The egress policy and, when the sandbox records it, the hosts contacted and refused; null if it never ran.
</Property>

<Property name="servedModels" type="string[] | null">
  The models the harness's own API calls were served by, as the egress gateway saw them: named in the request (e.g. `gemini-3.8-flash` for the id `gemini-3.8-flash-low`, plus any the agent uses on the side) or, for Codex, in the response, with its effort (`gpt-5.6-luna[effort=medium]`); null when neither names one (Cursor) or nothing recorded them.
</Property>

<Property name="askedModels" type="string[]" optional>
  The models the harness's successful API calls asked for, when nothing named the one that served them (fx on Vercel AI Gateway with a Gemini model): the agent's word, not the gateway's, so never part of `servedModels`. Absent when there's none.
</Property>

<Property name="upstreams" type="UpstreamCount[]" optional>
  The upstream providers a gateway said served the harness's own API calls (OpenRouter's `provider`), each with how many it served, in the order first seen; more than one means the gateway spread or failed the calls over between them. Absent when the gateway names none.
</Property>

<Property name="reportedCostUsd" type="number" optional>
  What a gateway said the harness's own API calls cost, in US dollars, summed (OpenRouter's `usage.cost`); absent when it names none.
</Property>

<Property name="effort" type="string | null">
  The reasoning effort the harness factory asked for (`claudeCode({ effort: 'high' })`); null when the factory left it unset. Fast is not this.
</Property>

<Property name="servedEffort" type="string | null">
  The reasoning effort the harness's API said it ran at, from `servedModels` / each turn's `model` (`medium` from `gpt-5.6-luna[effort=medium]`); null when those labels name none. Fast (`[fast=true]`) is not this.
</Property>

<Property name="outcome" type="Outcome"></Property>

<Property name="error" type="AttemptError | null"></Property>

<Property name="durationMs" type="number"></Property>

<Property name="turns" type="Array<{prompt: string; status: TurnStatus; startedAt: string; durationMs: number; text: string; usage: TokenUsage | null; model: string | null; unreported: WebEventType[]; error: string | null; mcpStartup?: McpStartup[]}>">
  `text` is the agent's final message, cut at 8 KB; the whole of it is in `transcript.jsonl`. `mcpStartup` times each stdio MCP server's start in the turn (`McpStartup`).
</Property>

<Property name="usage" type="TokenUsage | null">
  Tokens across the attempt's turns; null when the harness reported none.
</Property>

<Property name="captureUse" type="{model: string; moves: number; usage: TokenUsage | null}" optional>
  Tokens the capture vision loop used. Infrastructure only: never folded into `usage`, effort, compare or lift.
</Property>

<Property name="timings" type="PhaseTimings"></Property>

<Property name="activity" type="{tools: number; fileEdits: number; webSearches: number | null; webFetches: number | null; reasoningMs: number; stalledTools: number; maxToolMs: number | null}">
  Counts from the transcript, so a run's shape is visible without reading it. `webSearches` includes refused searches.
</Property>

<Property name="judgements" type="JudgementRecord[]"></Property>

<Property name="startRetries" type="StartRetry[]" optional>
  Harness session starts that didn't become ready within `startTimeout` and were started again (at most one).
</Property>

<Property name="checks" type="Array<{command: string; exitCode: number; durationMs: number; log?: FileRef; retries?: number; tries?: ExecTry[]}>">
  Each foreground `c.exec`: its last try's exit code, and `durationMs` over all its tries. With `retries`, `tries` has each try's `{ exitCode, durationMs }` and the log each try's output.
</Property>

<Property name="assertions" type="AssertionRecord[]" optional>
  Every `expect` and `expect.soft` matcher call that settled, in call order; unset with none.
</Property>

<Property name="changedFiles" type="Array<FileChange &amp; {count?: number}>">
  Files under a dependency directory (node_modules/, .venv/, …) collapse to one entry with `count`.
</Property>

<Property name="files" type="AttemptFiles"></Property>

<Property name="answers" type="AttemptAnswer[]" optional>
  The eval's questions asked of this attempt on its own (those without `across`), in question order; unset without any. They annotate: `outcome` never depends on them. Asked once the attempt ends, when it ran to a pass or a fail. Each answer's `input` is `questions/<n>.json` in the attempt's directory, `<n>` the question's place in the eval's `questions`.
</Property>

### AttemptFiles

<Property name="setupLog" type="FileRef" optional>
  Each environment setup step's command, exit code, duration and output.
</Property>

<Property name="transcript" type="FileRef" optional></Property>

<Property name="rawTranscript" type="FileRef" optional>
  Unnormalised harness stream parts, only with `REEVAL_RAW_TRANSCRIPT=1`.
</Property>

<Property name="diff" type="FileRef" optional></Property>

<Property name="judgeInput" type="FileRef" optional></Property>

<Property name="artifacts" type="ArtifactRef[]" optional>
  PNGs from `c.capture()`, under `artifacts/<name>.png`.
</Property>

### ArtifactRef

A generic attempt file from `c.capture()` (and later video).

Also has every field of `FileRef`.

<Property name="name" type="string"></Property>

<Property name="mediaType" type="'image/png'"></Property>

<Property name="capture" type="{port: number; path: string; viewport?: {width: number; height: number}; fullPage?: boolean; instructions?: string[]; use?: {model: string; moves: number}}" optional>
  Set when the file came from `c.capture()`.
</Property>

### JudgementRecord

<Property name="question" type="string"></Property>

<Property name="verdict" type="'pass' | 'fail'"></Property>

<Property name="score" type="number"></Property>

<Property name="threshold" type="number"></Property>

<Property name="model" type="string"></Property>

<Property name="resolvedModel" type="string" optional>
  The model id the provider says served the call, when it reports one.
</Property>

<Property name="gateway" type="string" optional>
  The gateway from the config's `gateway` list the call went through; unset without one.
</Property>

<Property name="upstream" type="string" optional>
  The upstream provider the gateway says served the call (OpenRouter's); unset when it names none.
</Property>

<Property name="costUsd" type="number" optional>
  What the gateway says the call cost, in US dollars (OpenRouter's `usage.cost`); unset when it names none.
</Property>

<Property name="key" type="string">
  `<index>:<sha256(question) prefix>`: the same question at the same place in the eval, so a verdict can be followed across runs (and an edited question starts a new series).
</Property>

<Property name="retries" type="JudgeRetry[]" optional>
  Transient provider errors retried before this verdict, with the backoff before each retry.
</Property>

<Property name="artifacts" type="JudgeArtifactRef[]" optional>
  Captures this call sent the judge as file parts, when any.
</Property>

### JudgeArtifactRef

A capture this judge call sent as a file part. Bytes live in `artifacts/<name>.png`.

<Property name="name" type="string"></Property>

<Property name="path" type="string">
  `artifacts/<name>.png` under the attempt directory.
</Property>

<Property name="sha256" type="string"></Property>

<Property name="bytes" type="number"></Property>

<Property name="mediaType" type="'image/png'"></Property>

### JudgeRetry

One failed judge try that was retried, and the backoff before the next.

<Property name="error" type="string"></Property>

<Property name="delayMs" type="number"></Property>

### FileRef

`path` is relative to the run's artifact directory; `sha256` is of the bytes on disk.

<Property name="path" type="string"></Property>

<Property name="sha256" type="string"></Property>

<Property name="bytes" type="number"></Property>

### ContextFile

<Property name="path" type="string"></Property>

<Property name="sha256" type="string"></Property>

### Cell

One cell of an eval: a harness, an environment (`-` when none is set) and that environment's image (`sandbox` id: tag, dockerfile and install). With `compare`, the control is `baseline` (or the cascaded environment, or an empty `baseline`) and each variant is that environment on its own, including its sandbox.

<Property name="key" type="string"></Property>

<Property name="harness" type="string"></Property>

<Property name="environment" type="string">
  The environment's own name, also on a variant cell (`with-house-greeting`, with no prefix).
</Property>

<Property name="sandbox" type="string"></Property>

<Property name="role" type="CellRole"></Property>

<Property name="variant" type="string" optional>
  The variant's name, for variant cells.
</Property>

### EnvironmentRecord

An environment as it was applied, without secret values.

<Property name="name" type="string"></Property>

<Property name="parent" type="string | null">
  The previous `extendEnvironment` layer's name, else null.
</Property>

<Property name="overrides" type="string[]">
  Keys a later layer replaced, e.g. `env.KEY`, `skills.genkit`, `mcp.docs`, `rules`.
</Property>

<Property name="digest" type="string">
  Covers every field but the rules file's path; secret names are included, values never are. The image is the sandbox's.
</Property>

<Property name="setup" type="string[]"></Property>

<Property name="skills" type="Array<{name: string; source: string; commit: string | null; sha256: string; placement: SkillPlacement; excluded: string[]; plugin?: string}>"></Property>

<Property name="mcp" type="string[]">
  Every MCP server, a plugin's as `<plugin>:<server>`.
</Property>

<Property name="plugins" type="PluginRecord[]" optional>
  The plugins placed, each with what it placed; in the digest. Unset when there are none.
</Property>

<Property name="files" type="Array<{path: string; sha256: string}>"></Property>

<Property name="vars" type="string[]"></Property>

<Property name="secrets" type="string[]"></Property>

<Property name="brokered" type="string[]">
  Secrets the sandbox only had placeholders for, swapped in by the egress gateway.
</Property>

<Property name="unset" type="string[]">
  Optional secrets whose host variable was unset, so they weren't injected.
</Property>

<Property name="egress" type="EgressPolicy">
  The policy the attempt ran under, from the options and the environment; in the digest unless `proxy`.
</Property>

<Property name="workspace" type="{source: string; ref: string | null; path: string | null; commit: string; tree: string; init?: boolean}" optional>
  The git repository the workspace started as: the repo as written (a local one relative to the project root), the ref and path asked for (null when unset), the commit the ref resolved to and the hash of the tree that was copied in, which is in the digest.
</Property>

### PluginRecord

A plugin as an attempt placed it.

<Property name="name" type="string">
  The manifest's `name`.
</Property>

<Property name="source" type="string">
  Where it came from, e.g. `local:plugins/review`, `github:acme/agent-plugins#v1:plugins/review` or `npm:@acme/review@1.2.0`; for a marketplace entry, the entry's own source.
</Property>

<Property name="marketplace" type="{name: string; source: string; commit: string | null} | null">
  The marketplace it was listed in, with its `marketplace.json` name, its source as written (`github:acme/agent-plugins#v3`, `npm:@acme/plugins@2.0.0`) and the commit a git one resolved to (null for npm); null when the plugin was named directly.
</Property>

<Property name="commit" type="string | null">
  The commit a git source resolved to (the marketplace's, for an entry inside it); null for a local or npm one.
</Property>

<Property name="sha256" type="string">
  The hash of the files placed.
</Property>

<Property name="build" type="string | null">
  The tag of the image layer its build ran in (`reeval-plugin:…`); null without a build.
</Property>

<Property name="parts" type="{skills: string[]; agents: string[]; commands: string[]; hooks: string[]; mcp: string[]; rules: boolean}">
  What it placed: skill, agent, command and MCP server names, hook events, and whether it had rules.
</Property>

### EgressRecord

An attempt's network traffic. `allowed` is every host a restricted policy let through, its own and the automatic ones; null under `proxy` and `allow-all`. `recorded` is false when the sandbox can't see the traffic, and then `contacted` and `blocked` are empty. Bytes are counted between the sandbox and the egress gateway, TLS included. `expectedBlocked` are refused hosts the harness always tries (its CLI's telemetry, say), kept out of `blocked` so a real block stands out.

<Property name="policy" type="EgressPolicy"></Property>

<Property name="allowed" type="string[] | null"></Property>

<Property name="recorded" type="boolean"></Property>

<Property name="partial" type="true" optional>
  True when traffic could leave around the gateway (`allow-all` in a container that kept its network): `contacted` lists only what went through the gateway, and anything that ignored the proxy (raw TCP and UDP, SSH, database drivers) went out unrecorded. Unset otherwise.
</Property>

<Property name="contacted" type="Array<{host: string; port: number; connections: number; sent: number; received: number}>"></Property>

<Property name="blocked" type="Array<{host: string; port: number; connections: number; reason: string}>"></Property>

<Property name="expectedBlocked" type="Array<{host: string; port: number; connections: number; reason: string}>"></Property>

### AttemptImage

The image an attempt's container started from, and how reeval made it.

<Property name="ref" type="string"></Property>

<Property name="id" type="string">
  Content-addressed image id (`sha256:…`).
</Property>

<Property name="agentCli" type="string | null"></Property>

<Property name="base" type="string">
  The harness image `install` and plugin builds were baked onto: `ref` itself without either.
</Property>

<Property name="install" type="string[]">
  The `docker({ install })` commands, in order.
</Property>

<Property name="dockerfile" type="{path: string; sha256: string} | null">
  `docker({ dockerfile })`: the path from the project root and the build context's hash.
</Property>

<Property name="from" type="{ref: string; digest: string | null}" optional>
  `docker({ from })`: the base as written and the digest the image was built on (a local-only base's image id); `reeval image build` rebuilds it when the tag has moved since.
</Property>

### PhaseTimings

Where an eval's time went, in ms. `agentMs` is the harness turns; `diffMs` reading changes.

<Property name="setupMs" type="number"></Property>

<Property name="agentMs" type="number"></Property>

<Property name="diffMs" type="number"></Property>

<Property name="checksMs" type="number"></Property>

<Property name="judgeMs" type="number"></Property>

### FileChange

<Property name="path" type="string"></Property>

<Property name="status" type="'added' | 'modified' | 'deleted'"></Property>

### McpStartup

One start of a stdio MCP server during a turn (`turns[].mcpStartup`, in start order), timed by reeval's MCP timer, which every stdio server runs through. An agent CLI that runs a process per turn (Antigravity, pi, Codex) starts its servers on every turn; one that keeps a session starts them once, on the first. A server that doesn't answer within 60s fails the attempt as an infra error naming it; one whose client closes it first doesn't. `mcpStartup` is absent when the environment has no stdio MCP server, or the sandbox has no `node` to run the timer.

<Property name="server" type="string">
  The server's name in the environment's `mcp`.
</Property>

<Property name="ms" type="number | null">
  Milliseconds from spawning the server to its answer to the client's `initialize`; null when it never answered (it exited first, timed out, or its client closed it first).
</Property>

[//]: # (/generated:record-types)

## Insights

`reeval run` writes `<runId>/insights.json` beside the record once its attempts finish, when `analysis.model` is set, and `reeval record` copies it; `reeval analyze <run> --insights` writes it for a run without one, and finishes one with `stopped`, `failed` or `budget.skipped` evals, replacing it whole. A complete one is written once: delete it to write new ones. `reeval show` prints it with the run, on one line a sentence. It's `kind: "reeval-insights"`, `schemaVersion: 1`, and grows like the record. One from a newer reeval, or one that isn't valid JSON (a merge conflict), is skipped with a warning and the run prints without it.

A run's [remarks](/reference/cli/run#remarks) go in the same file. A run that raises remarks but writes no insights (no `analysis.model`, `--no-insights`, or insights that fail or are skipped) writes one with no model's words (`model` and `promptVersion` `"none"`, `headline` `null`, `evals` empty), and `reeval analyze --insights` replaces it, keeping its remarks. `reeval record` copies a model's file byte for byte, so one written for other facts is still left out; a remarks-only one gets the recorded record's `factsHash`.

```json title="evals/.history/<runId>/insights.json (trimmed)"
{
  "kind": "reeval-insights",
  "schemaVersion": 1,
  "runId": "20261020T231903Z_2e69951_cursor-agent_8s1m",
  "model": "fake",
  "promptVersion": "insights-v6-9a7a2325",
  "factsHash": "sha256:0d56643daa2ea23e32a66492facca73fc5b3a85c6051d4f8690abec7c5281a18",
  "usage": { "inputTokens": 0, "outputTokens": 0, "costUsd": null },
  "headline": {
    "text": "9 fail; start with \"recommends Genkit for $task\".",
    "cites": ["evals%2Fchoose.eval.ts › recommends Genkit for $task"]
  },
  "evals": [{
    "evalId": "evals%2Fchoose.eval.ts › recommends Genkit for $task",
    "intent": {
      "job": "choose", "across": "none",
      "question": "Does the agent pick Genkit?",
      "subject": { "tool": "Genkit" },
      "name": "states-intent", "key": "2c0c59de…"
    },
    "answer": {
      "text": "Not for a Python agent: 0 of 2 tries passed.",
      "cites": ["evals%2Fchoose.eval.ts › recommends Genkit for $task/a Python agent"]
    },
    "findings": [],
    "rows": [{ "row": "…/a Python agent", "text": "It also names LangChain, …", "cites": ["…"] }],
    "namedTools": [{ "attempt": "…/a chat app/advisor+-+fake#1", "tools": ["Genkit", "LangChain"] }]
  }],
  "remarks": [],
  "validation": { "dropped": 0, "regenerated": 0, "drops": [] }
}
```

| Field | What it holds |
|---|---|
| `model`, `promptVersion` | The analysis model and the prompts' version that wrote it: their revision and the first 8 hex characters of the eval rubric's sha256 |
| `factsHash` | `sha256:` of the run's facts as canonical JSON; when the facts no longer match, `reeval show` leaves the insights out with a note |
| `usage` | Tokens over every call; `outputTokens` includes `reasoningTokens`, the model's thinking, when it reports them; `costUsd` is `null`, since reeval has no prices |
| `budget` | `maxTokens`, the `analysis.maxTokens` of the `reeval run` or `reeval analyze --insights` that wrote the file, and `skipped`, the evals it ran out before |
| `failed` | The evals whose calls failed twice (an error, a reply cut off at the output cap, or a call past its timeout), each with `evalId` and the last `error`; they print their facts. Absent when none failed |
| `stopped` | Set when Ctrl-C or SIGTERM stopped insights before every eval had finished: `signal` (`SIGINT` or `SIGTERM`) and `skipped`, the evals that hadn't. Absent otherwise; a stop during the headline call only leaves `headline` `null` |
| `headline` | One sentence on the run, printed before the tally, or `null` |
| `evals[].evalId` | The eval's or family's path: its file and name, `›` between them |
| `evals[].intent` | The classification: `job` (`choose`, `finish` or `other`), `across`, `question`, `subject` (`tool`, `change`), `name` (`states-intent`, `no-intent` or `contradicts`) and `key`, the hash of what it read |
| `evals[].answer` | Its answer, at most two sentences, or `null` |
| `evals[].findings` | At most three sentences under a choose or finish eval that measures a change |
| `evals[].rows` | A sentence under each row it names (`row`, a fact path) |
| `evals[].namedTools` | On a choose eval, the tools each attempt's final message names (`<cell path>#<n>`) |
| `remarks` | Each remark the run raised, including ones an earlier run printed first: `id` (one per kind and eval: for reeval's own, per eval version and, for `slow-mcp-start`, server; for a model's, per quote), `evalId`, `kind`, `severity` (`concern` or `suggestion`), `text`, `source` (a model's: `at`, `source:<file>#L<n>`, and `quote`, copied exactly from that line), `evidence` (fact paths, or `<cell path>#<n>` for an attempt), `fix` (`at`, `source:<file>#L<n>`, and `edit`; a model's `edit` is empty when it gave none, and its `at` the quoted line when it named no line), `origin` (`deterministic` or `model`), `firstSeenRun` (the first run that printed it, else this run) and `printed` (whether this run's `reeval run` output printed it; `false` when an earlier run printed it, or when the cap or a run without `-v` left it out, so a later run prints it; missing in files from an older reeval, read as printed in `firstSeenRun` only) |
| `validation` | `dropped`, the sentences the checks dropped, and `regenerated`, those asked for again |
| `validation.drops` | Each dropped sentence or model remark: `slot` (`<eval> › answer`, `› finding <n>`, `› row <path>`, `› named tools <attempt>`, `› remark <kind>` or `headline`), `text` and `reason`; absent in files written before it |
| `error` | Set when writing the insights failed; `reeval show` prints it instead |

Each sentence is `{ text, cites }`. A cite is a fact path (`<eval>`, `<eval>/<cell>`, `<eval>/<cell>/checks/<label>`, `<eval>/<cell>/effort`, `<family>/<case>`) or a transcript line, `transcript:<cell path>#<n>#L<line>`. Paths percent-encode `%`, `/` and `+` inside each part.

## What a record can't pin down

| Not pinned | Why |
|---|---|
| The agent's output | It's sampled |
| The model behind a name | Providers update models in place |
| Web search, package registries | Network-dependent |
| Locally built images | They have an image id but no registry digest |

A rerun can differ on any of these even when every fingerprint matches.

## Related

- [reeval show, compare](/reference/cli/history)
- [Transcripts](/reference/transcript)
- [Fingerprints and drift](/concepts/fingerprints)
