---
title: Run record reference
description: >-
  A run record holds one run's cells, attempts, assertions, judgments, and the
  environment each ran in.
---
> ## Documentation Index
> Fetch the complete documentation index at: https://docs.reeval.ai/llms.txt
> Use this file to discover all available pages before exploring further.

The `reeval show --json` command prints one run's record. The record is the `<runId>.json` file in the `.reeval/runs/` directory, or the same name in the history folder, with artifacts in the `<runId>` directory.

For more information about saving a record, see [Recording](/evaluations/runs/recording). For more information about the saved result, see [Record](/reference/glossary#record). For more information about querying recorded runs, see [Query views](/reference/query-views).

## Value types

Each row is one closed alias. `RecordMode` is the `record` option.

[//]: # (generated:record-values)

| Type | Values |
|---|---|
| `Outcome` | `'pass'`, `'fail'`, `'infra_error'`, `'skipped'` |
| `ErrorKind` | `'assertion'`, `'timeout'`, `'aborted'`, `'infra'`, `'judge'` |
| `TurnStatus` | `'completed'`, `'aborted'`, `'error'` |
| `ArtifactLevel` | `'all'`, `'failures'`, `'none'` |
| `RecordMode` | `'manual'`, `'ci'`, `'always'` |
| `FamilyVerdict` | `'lift'`, `'drop'`, `'noise'`, `'insufficient'` |

[//]: # (/generated:record-values)

`CellRole` is `control` or `variant`. `CellGate` is `all`, `{ passRate: number }`, or `none`. `CheckVerdict` is `discriminates`, `both-pass`, `both-fail`, or `mixed`. `EgressPolicy` is `proxy`, `allow-all`, `deny-all`, or `{ allow: string[] }`. `SkillPlacement` is `harness` or `workspace`. `WebEventType` is `web-search` or `web-fetch`. `CompareCut` is `within` or `across`.

For more information about which hosts a sandbox can reach, see [Egress](/configuration/environments/egress).

## Record types

`kind` is `reeval-run`, and `schemaVersion` is `1`. `git` is null outside a Git repository, or when that repository has no commits. `author` is `ci` on a CI run, and otherwise a slug of the Git user name. `config.concurrency` is how many attempts run together, and `config.configHash` omits it.

For more information about the keys on `stack.judge.state`, see [the `judge` option](/reference/options#judge).

[//]: # (generated:record-types)

### `RunRecord`

<Property name="kind" type="'reeval-run'"></Property>

<Property name="schemaVersion" type="typeof SCHEMA_VERSION"></Property>

<Property name="runId" type="string">
  Also the file name: `<UTC start>_<short sha>_<author or ci>`.
</Property>

<Property name="status" type="'complete' | 'interrupted' | 'budget'">
  `budget` when `budget.duration` stopped new attempts from starting.
</Property>

<Property name="startedAt" type="string"></Property>

<Property name="endedAt" type="string"></Property>

<Property name="durationMs" type="number"></Property>

<Property name="author" type="string"></Property>

<Property name="git" type="GitInfo | null"></Property>

<Property name="runner" type="RunnerInfo"></Property>

<Property name="dependencies" type="{lockfiles: Array<{path: string; sha256: string}>}"></Property>

<Property name="stack" type="Stack"></Property>

<Property name="environments" type="EnvironmentRecord[]">
  Each environment the run applied, fingerprinted; secret values are never included.
</Property>

<Property name="profile" type="{name: string; reason: string; defaults: {harness?: string[]; environment?: string; sandbox?: string; runs?: number; timeout?: string | number; env?: string[]}; limit: {runs?: number; compare?: false | {maxVariants: number}; harness?: string[]}} | null">
  The profile the run was sized with, and why it was chosen.
</Property>

<Property name="plan" type="{evals: number; cells: number; attempts: number; estimateMs: number; sandboxWarnings: SandboxWarning[]}">
  What the run planned before it started, and each level whose sandbox ran as root or with capabilities added back.
</Property>

<Property name="budget" type="{attempts?: number; duration?: string | number} | null"></Property>

<Property name="config" type="{sandbox: string; environment: string | null; runs: number; judgeModel: string | null; threshold: number; timeoutMs: number; concurrency: number; configHash: string}">
  The run's settings. `sandbox` is the id of each sandbox the planned attempts ran on, after the profile's defaults, joined with `, `.
</Property>

<Property name="cost" type="Cost"></Property>

<Property name="artifacts" type="ArtifactLevel">
  Which artifacts sit next to this record. Refs to files that weren't kept are removed.
</Property>

<Property name="redaction" type="{version: 1}"></Property>

<Property name="summary" type="RunSummary"></Property>

<Property name="evals" type="EvalEntry[]"></Property>

<Property name="families" type="FamilyRecord[]" optional>
  Each pooled `evaluate.each` that ran; unset when none did.
</Property>

<Property name="errors" type="string[]" optional>
  These are redacted errors outside any attempt: an evaluation file that didn't load, a rejection or throw no attempt awaited, or a worker that died. Any one fails the run. Unset when there were none.
</Property>

### `GitInfo`

<Property name="sha" type="string">
  The commit the run executed at: HEAD when `reeval run` started. The later commit that adds the history file is a different, descendant commit.
</Property>

<Property name="branch" type="string | null"></Property>

<Property name="dirty" type="boolean">
  Uncommitted changes, outside the `.reeval/` directory and the history folder, when the run started.
</Property>

<Property name="diffHash" type="string | null">
  sha256 of the uncommitted changes (tracked and untracked, as a binary patch), when dirty.
</Property>

<Property name="worktreePatch" type="FileRef | null">
  Those changes are stored as the `worktree.patch` file in the artifact directory. Applying it to `sha` reproduces the workspace.
</Property>

<Property name="worktreePatchNote" type="string | null">
  This explains why a dirty run has no `worktree.patch` file: it was too large, `artifacts` was `'none'`, or it looked like it held a secret (then where, and which rule, never the value).
</Property>

<Property name="commitTime" type="string"></Property>

<Property name="shallow" type="boolean"></Property>

<Property name="remote" type="string | null">
  The `origin` URL with any credentials removed.
</Property>

### `Stack`

These fields describe what ran and graded the agents: the machine, images, harnesses, and judge.

<Property name="reeval" type="string">
  The reeval version that wrote the run.
</Property>

<Property name="node" type="string">
  `process.versions.node`, or the compatible Node version reported by another runtime.
</Property>

<Property name="runtime" type="{name: string; version: string}" optional>
  The runtime reeval ran on, `node`, `bun`, `deno`, or another, and its version.
</Property>

<Property name="language" type="{name: string; version?: string}" optional>
  The evaluation language, `typescript` for `*.eval.ts` files, and the version of its toolchain when the runner reports one.
</Property>

<Property name="os" type="string"></Property>

<Property name="arch" type="string"></Property>

<Property name="images" type="Array<AttemptImage &amp; {repoDigests: string[]}>">
  Every image an attempt started from. `id` is content-addressed; `repoDigests` exist only after the image is pushed.
</Property>

<Property name="packages" type="Record<string, string>">
  Exact versions of the packages that run and grade the agent.
</Property>

<Property name="harnesses" type="Array<{id: string; kind: string; model: string | null; effort: string | null; resolvedModels: string[]; resolvedEffort: string[]}>">
  For each harness, the record includes its configured `model` and requested `effort`, `resolvedModels` (what it reported serving, with the variant it resolved), and `resolvedEffort` (reasoning-effort suffixes those labels named). Fast (`[fast=true]`) is not `resolvedEffort`.
</Property>

<Property name="judge" type="{model: string | null; resolvedModels: string[]; threshold: number; version: string; state?: ResolvedJudgeStateOptions}">
  The run's judge: the config's, with what the profile's `judge` and `defaults.judge` set over it. `config.judgeModel` and `config.threshold` name the same one.
</Property>

<Property name="context" type="ContextFile[]">
  Union of every attempt's context files.
</Property>

<Property name="fingerprintScope" type="string[]">
  What evaluation fingerprints hash; fingerprints from different scopes aren't comparable.
</Property>

### `RunSummary`

Counts attempts, except `evals`.

<Property name="evals" type="number"></Property>

<Property name="attempts" type="number"></Property>

<Property name="pass" type="number"></Property>

<Property name="fail" type="number">
  Every failed attempt, including `notGated` ones.
</Property>

<Property name="notGated" type="number">
  Failed attempts in ungated cells (controls, by default), which don't fail the run.
</Property>

<Property name="infraError" type="number"></Property>

<Property name="skipped" type="number"></Property>

### `SandboxWarning`

A `SandboxWarning` describes a sandbox with more privilege than the default `reeval` user: `docker({ root: true })` or `docker({ capabilities })`. One per level that set such a sandbox, however many evaluations inherit it.

<Property name="kind" type="'root' | 'capabilities'"></Property>

<Property name="level" type="'config' | 'profile' | 'group' | 'eval'">
  Which level set the sandbox.
</Property>

<Property name="where" type="string">
  The value is the level as reeval names it: the `reeval.config.ts` file, `profiles.pr.defaults`, `group("net")`, or `evaluate("installs deps")`.
</Property>

<Property name="file" type="string | null">
  The evaluation file that sets it, for a group or evaluation; null for the config or a profile.
</Property>

<Property name="capabilities" type="string[]" optional>
  The capabilities added back, for `capabilities`.
</Property>

<Property name="message" type="string">
  The line reeval prints, after `WARNING: ` (root) or `warning: ` (capabilities).
</Property>

### `Cost`

The tokens the run used, as the harnesses count them; reeval prices nothing.

<Property name="inputTokens" type="number | null">
  Prompt tokens, cache reads and writes included.
</Property>

<Property name="outputTokens" type="number | null"></Property>

<Property name="usage" type="TokenUsage | null">
  The same totals split as each turn reports them.
</Property>

<Property name="source" type="'reported' | 'unreported'"></Property>

### `EvalEntry`

<Property name="id" type="string">
  `file › group › … › name`, as shown.
</Property>

<Property name="key" type="string">
  Used to match an evaluation across runs; equal to `id`.
</Property>

<Property name="slug" type="string">
  Directory name inside the artifact directory, from `key`.
</Property>

<Property name="file" type="string"></Property>

<Property name="group" type="string[]"></Property>

<Property name="name" type="string"></Property>

<Property name="fingerprint" type="string">
  Results are comparable only when fingerprints match.
</Property>

<Property name="sources" type="Array<{path: string; sha256: string}>">
  The evaluation file, by content hash: where the evaluation lives. Records from before `source` also list the local files it imported, and were fingerprinted over them.
</Property>

<Property name="source" type="EvalSource" optional>
  Hashes of what the `sources` fingerprint part covers but `revision`: the eval's body, its case data and its options with its groups'. Unset on records from before it.
</Property>

<Property name="revision" type="Revision" optional>
  The eval's `revision`, part of the `sources` fingerprint part; unset when it sets none.
</Property>

<Property name="tags" type="string[]"></Property>

<Property name="compare" type="EvalCompareMeta" optional>
  The baseline environment, when the evaluation uses `compare`.
</Property>

<Property name="outcome" type="Outcome">
  Over gated cells, the outcome is `fail` if any failed, else `infra_error` if any is (a cell is `infra_error` only when nothing counted), else `skipped` if all were, else `pass`. An ungated cell's failures don't fail the evaluation.
</Property>

<Property name="todo" type="boolean">
  A placeholder, `evaluate.todo` or `group.todo`: it never runs, and its outcome is `skipped`.
</Property>

<Property name="passRate" type="number | null">
  pass@1, pass / (pass + fail), over the gated cells' attempts; infra errors and skips excluded.
</Property>

<Property name="cells" type="CellRecord[]"></Property>

<Property name="attempts" type="AttemptRecord[]"></Property>

<Property name="answers" type="CompareAnswer[]" optional>
  The eval's pairwise questions (those with `across`), answered after its attempts by votes, one per question; unset without any. They annotate: `outcome` and `passRate` never depend on them. Each vote may include a short `reason` (capped). `reeval analyze` includes each answer in the digest (rates by cut, votes, ties, a failed call, and a sample of reasons). Each answer's `input` is the `<eval-slug>/questions/<n>.json` file: every call its model was sent, per pair and order, with secrets masked and screenshots as refs; unset where the evaluation's files weren't kept.
</Property>

<Property name="family" type="{id: string; name: string; case: number; cases: number}" optional>
  Set on each case of a pooled `evaluate.each`; `RunRecord.families` has the suite.
</Property>

### `CellRecord`

One harness and environment of an evaluation, and how its attempts went.

Also has every field of `Cell`.

<Property name="fingerprint" type="string">
  `sha256` of `fingerprintParts`; results are comparable only when it matches.
</Property>

<Property name="fingerprintParts" type="FingerprintParts"></Property>

<Property name="environmentDigest" type="string | null">
  The environment's digest; null for none. With `environment`, the key into `RunRecord.environments`: environments with the same contents share a digest.
</Property>

<Property name="harnessKind" type="string"></Property>

<Property name="gateway" type="string" optional>
  The gateway in the config's `gateway` list its harness resolved to (`direct` is each provider's own key); unset when no config, profile or pin named one.
</Property>

<Property name="model" type="string | null">
  The configured harness model, when it has one.
</Property>

<Property name="effort" type="string | null">
  The reasoning effort the harness factory asked for; null when it left it unset. Fast is not this.
</Property>

<Property name="gate" type="CellGate">
  How the attempts decide the cell's outcome; `'none'` when it doesn't fail the evaluation.
</Property>

<Property name="overrides" type="string[]">
  Environment keys the variant replaced on the control's (or a child on its parent's).
</Property>

<Property name="runs" type="number"></Property>

<Property name="pass" type="number"></Property>

<Property name="fail" type="number"></Property>

<Property name="infraError" type="number"></Property>

<Property name="skipped" type="number"></Property>

<Property name="passRate" type="number | null">
  pass@1: pass / (pass + fail); infra errors and skips excluded.
</Property>

<Property name="outcome" type="Outcome">
  By `gate`. `'all'` (pass^k: every counted attempt passes) matches `{ passRate: 1 }`; `{ passRate }` is a pass@1 threshold. Infra only when nothing counted; `'none'` reports the worst outcome. A variant below its `minLift` fails too.
</Property>

<Property name="lift" type="Lift" optional></Property>

<Property name="checks" type="CellCheck[]" optional>
  Each assertion, by label, seen in any attempt of the evaluation, over this cell's counted attempts; on a variant cell, the control's beside it. Unset when no attempt recorded any.
</Property>

### `Lift`

A variant cell's pass@1 against the control's, on the same harness and sandbox.

<Property name="vs" type="string">
  The control cell's key.
</Property>

<Property name="control" type="{pass: number; counted: number}">
  Passes out of counted attempts (passes and fails) on each side.
</Property>

<Property name="variant" type="{pass: number; counted: number}"></Property>

<Property name="points" type="number">
  Variant pass@1 minus control pass@1, from -1 to 1.
</Property>

<Property name="interval" type="{lo: number; hi: number; level: 0.95; method: 'newcombe-wilson'}">
  95% interval on the difference, by Newcombe's hybrid score method from two Wilson intervals.
</Property>

<Property name="withinNoise" type="boolean">
  The interval spans 0: the attempts can't tell the variant from the control.
</Property>

<Property name="minLift" type="number" optional>
  Set when the compare has a `minLift`; the cell fails when `interval.lo` is below it.
</Property>

### `FamilyRecord`

A pooled `evaluate.each`: its cases as samples of one suite. Each case is also an evaluation.

<Property name="id" type="string">
  `file › group › … › template`.
</Property>

<Property name="file" type="string"></Property>

<Property name="group" type="string[]"></Property>

<Property name="name" type="string">
  The name template, for example `fixes $id`.
</Property>

<Property name="cases" type="number">
  Cases in the list.
</Property>

<Property name="ran" type="number">
  Cases with an evaluation in this run; fewer when a filter or tag left some out.
</Property>

<Property name="rates" type="FamilyRate[]">
  One per cell.
</Property>

<Property name="lifts" type="FamilyLift[]">
  One per variant cell, with `compare`.
</Property>

<Property name="outcome" type="'pass' | 'fail'">
  `fail` when a lift is below its `minLift`. The cases still pass or fail on their own.
</Property>

### `FamilyRate`

One cell's pass@1 over a family's cases.

<Property name="cell" type="string">
  The cell's key, the same in every case.
</Property>

<Property name="cases" type="number">
  Cases with at least one counted attempt in this cell.
</Property>

<Property name="pass" type="number">
  Passes and counted attempts over those cases.
</Property>

<Property name="counted" type="number"></Property>

<Property name="rate" type="number">
  The mean of the cases' pass@1; at one attempt a case, `pass / counted`.
</Property>

<Property name="interval" type="FamilyInterval"></Property>

### `FamilyLift`

A variant cell against its control over a family's cases, paired by case.

<Property name="variant" type="string">
  The variant cell's key.
</Property>

<Property name="vs" type="string">
  The control cell's key.
</Property>

<Property name="pairs" type="number">
  Cases with counted attempts on both sides.
</Property>

<Property name="variantRate" type="number">
  Mean pass@1 over the paired cases on each side.
</Property>

<Property name="controlRate" type="number"></Property>

<Property name="discordant" type="{variantOnly: number; controlOnly: number}" optional>
  Set at one attempt a case and side: pairs where only one side passed.
</Property>

<Property name="points" type="number">
  Variant minus control, from -1 to 1.
</Property>

<Property name="interval" type="FamilyInterval"></Property>

<Property name="p" type="number | null">
  McNemar's exact p (one attempt a case), else Wilcoxon signed-rank over the cases.
</Property>

<Property name="withinNoise" type="boolean">
  The interval spans 0.
</Property>

<Property name="verdict" type="FamilyVerdict"></Property>

<Property name="minLift" type="number" optional>
  The compare's `minLift`: on a family it gates this lower bound, not each case's.
</Property>

<Property name="outcome" type="'pass' | 'fail'">
  `fail` when `interval.lo` is below `minLift`.
</Property>

<Property name="effort" type="FamilyEffort" optional></Property>

### `FamilyInterval`

A 95% interval on a pooled rate or difference.

<Property name="lo" type="number"></Property>

<Property name="hi" type="number"></Property>

<Property name="level" type="0.95"></Property>

<Property name="method" type="'wilson' | 'newcombe-paired' | 't-cases'">
  `wilson` or `newcombe-paired`: one counted attempt a case and side, so each case is one Bernoulli trial (or pair). `t-cases`: several, so a t interval over the cases' pass@1, with attempts within a case never counted as independent.
</Property>

### `FamilyEffort`

The variant's effort against the control's, paired by case.

<Property name="tools" type="FamilyEffortMetric"></Property>

<Property name="agentMs" type="FamilyEffortMetric"></Property>

<Property name="tokens" type="FamilyEffortMetric" optional>
  Over the cases where every counted attempt reported usage; omitted when none did.
</Property>

### `FamilyEffortMetric`

One effort metric over a family's paired cases.

<Property name="pairs" type="number">
  Cases with a value on both sides.
</Property>

<Property name="ratio" type="number | null">
  The median over cases of the variant's per-case median, over the control's; null at 0.
</Property>

<Property name="p" type="number | null">
  Wilcoxon signed-rank over the per-case differences; null when every difference is 0.
</Property>

<Property name="verdict" type="'fewer' | 'more' | 'noise' | 'insufficient'">
  `insufficient` below 6 non-zero differences, where the exact test can't reach 0.05. `fewer` and `more` are the test's direction (which side's differences rank higher), not the median's.
</Property>

### `FingerprintParts`

Each input of a cell fingerprint, hashed on its own, so a change can be named.

<Property name="sources" type="string">
  Over `EvalEntry.source` and `EvalEntry.revision`; on records from before them, over `EvalEntry.sources`.
</Property>

<Property name="harness" type="string">
  Over the harness descriptor, less concurrency caps.
</Property>

<Property name="environment" type="string | null"></Property>

<Property name="image" type="string | null"></Property>

<Property name="agentCli" type="string | null"></Property>

<Property name="judge" type="string">
  Over the judge's model, threshold, and version, and the config's `state` when it sets one.
</Property>

<Property name="context" type="string">
  Over the attempt's context files.
</Property>

<Property name="gateway" type="string" optional>
  The gateway its model calls went through; unset on `direct`, each provider's own key.
</Property>

<Property name="gate" type="string" optional>
  The cell's gate (`none`, `passRate 0.5`); unset on the default `all`. Compare reads a change in it as a policy change, not a reason the runs can't be compared.
</Property>

### `AttemptRecord`

<Property name="key" type="string">
  `<eval key> # <cell key>#<n>`: unique in the run.
</Property>

<Property name="id" type="string">
  `<cell key>#<n>`: unique in its evaluation.
</Property>

<Property name="cell" type="string">
  The cell's key; see `EvalEntry.cells`.
</Property>

<Property name="n" type="number">
  Run number within the cell, from 1.
</Property>

<Property name="seq" type="number | null">
  Start order across the run, from 1; null for attempts that never started.
</Property>

<Property name="startedAt" type="string | null"></Property>

<Property name="imageId" type="string | null">
  The image the container started from, in `stack.images`; null when the sandbox doesn't start from an image.
</Property>

<Property name="env" type="{vars: string[]; secrets: string[]; brokered: string[]}">
  Names of the variables and secrets set in the sandbox, and the brokered ones; never values.
</Property>

<Property name="context" type="ContextFile[]">
  Skills, rules, and MCP config visible to the agent in its sandbox, by content hash.
</Property>

<Property name="egress" type="EgressRecord | null">
  The egress policy and, when the sandbox records it, the hosts contacted and refused; null if it never ran.
</Property>

<Property name="servedModels" type="string[] | null">
  The models served by the harness's API calls, as observed by the egress gateway. Values may come from requests or responses and include reported effort; null when no model was named or recorded. The gateway reads the request URL for Antigravity, Goose, pi, and OpenCode on a `google/` model, so a request for `<model>-low` records `<model>`, and the API response for Codex, Grok Build, and OpenCode on another provider. Null for Cursor and Claude Code, whose API requests name no model.
</Property>

<Property name="askedModels" type="string[]" optional>
  The models the harness's successful API calls asked for, when nothing named the one that served them (fx on Vercel AI Gateway with a Gemini model): the agent's word, not the gateway's, so never part of `servedModels`. Absent when there's none.
</Property>

<Property name="upstreams" type="UpstreamCount[]" optional>
  The upstream providers a gateway said served the harness's own API calls (OpenRouter's `provider`), each with how many it served, in the order first seen; more than one means the gateway spread or failed the calls over between them. Absent when the gateway names none.
</Property>

<Property name="reportedCostUsd" type="number" optional>
  What a gateway said the harness's own API calls cost, in US dollars, summed (OpenRouter's `usage.cost`); absent when it names none.
</Property>

<Property name="effort" type="string | null">
  The reasoning effort the harness factory asked for (`claudeCode({ effort: 'high' })`); null when the factory left it unset. Fast is not this.
</Property>

<Property name="servedEffort" type="string | null">
  The reasoning effort the harness's API said it ran at, from `servedModels` or each turn's `model` (`medium` from `gpt-5.6-luna[effort=medium]`); null when those labels name none. Fast (`[fast=true]`) is not this.
</Property>

<Property name="outcome" type="Outcome"></Property>

<Property name="error" type="AttemptError | null"></Property>

<Property name="durationMs" type="number"></Property>

<Property name="turns" type="Array<{prompt: string; status: TurnStatus; startedAt: string; durationMs: number; text: string; usage: TokenUsage | null; model: string | null; unreported: WebEventType[]; error: string | null; mcpStartup?: McpStartup[]}>">
  `text` is the agent's final message, cut at 8 KB; the whole of it is in the `transcript.jsonl` file. `mcpStartup` times each stdio MCP server's start in the turn (`McpStartup`).
</Property>

<Property name="usage" type="TokenUsage | null">
  Tokens across the attempt's turns; null when the harness reported none.
</Property>

<Property name="captureUse" type="{model: string; moves: number; usage: TokenUsage | null}" optional>
  Tokens the capture vision loop used. Infrastructure only: never included in `usage`, effort, compare, or lift.
</Property>

<Property name="timings" type="PhaseTimings"></Property>

<Property name="activity" type="{tools: number; fileEdits: number; webSearches: number | null; webFetches: number | null; reasoningMs: number; stalledTools: number; maxToolMs: number | null}">
  Counts from the transcript, so the attempt's activity is available without reading the transcript. `webSearches` includes refused searches.
</Property>

<Property name="judgements" type="JudgmentRecord[]"></Property>

<Property name="startRetries" type="StartRetry[]" optional>
  Harness session starts that didn't become ready within `startTimeout` and were started again (at most one).
</Property>

<Property name="setupProcesses" type="SetupProcess[]" optional>
  Processes the setup steps left running in a Docker sandbox with a `to: 'checks'` secret without `hosts` in their environment, so they hold its real value while the agent works and the agent, as the same user, can read it from them. Found by the secret's variable name, so a copy under another name or in a command's arguments isn't. At most 20, each `{ command, secrets }`: its command line, masked and cut at 200 characters, and those secrets' names. Absent when there were none.
</Property>

<Property name="checks" type="Array<{command: string; exitCode: number; durationMs: number; log?: FileRef; retries?: number; tries?: ExecTry[]}>">
  Each foreground `c.exec`: its last try's exit code, and `durationMs` over all its tries. With `retries`, `tries` has each try's `{ exitCode, durationMs }` and the log each try's output.
</Property>

<Property name="assertions" type="AssertionRecord[]" optional>
  Every `expect` and `expect.soft` matcher call that settled, in call order; unset with none.
</Property>

<Property name="changedFiles" type="Array<FileChange &amp; {count?: number}>">
  Files under a dependency directory, such as the `node_modules/` directory or `.venv/` directory, collapse to one entry with `count`.
</Property>

<Property name="files" type="AttemptFiles"></Property>

<Property name="answers" type="AttemptAnswer[]" optional>
  The eval's questions asked of this attempt on its own (those without `across`), in question order; unset without any. They annotate: `outcome` never depends on them. Asked once the attempt ends, when it ran to a pass or a fail. Each answer's `input` is `questions/<n>.json` in the attempt's directory, `<n>` the question's place in the eval's `questions`.
</Property>

### `AttemptFiles`

<Property name="setupLog" type="FileRef" optional>
  Each environment setup step's command, exit code, duration, and output.
</Property>

<Property name="transcript" type="FileRef" optional></Property>

<Property name="rawTranscript" type="FileRef" optional>
  Unnormalized harness stream parts, only with `REEVAL_RAW_TRANSCRIPT=1`.
</Property>

<Property name="diff" type="FileRef" optional></Property>

<Property name="judgeInput" type="FileRef" optional></Property>

<Property name="artifacts" type="ArtifactRef[]" optional>
  PNGs from `c.capture()`, under the `artifacts/<name>.png` path.
</Property>

### `ArtifactRef`

A generic attempt file from `c.capture()`.

Also has every field of `FileRef`.

<Property name="name" type="string"></Property>

<Property name="mediaType" type="'image/png'"></Property>

<Property name="capture" type="{port: number; path: string; viewport?: {width: number; height: number}; fullPage?: boolean; instructions?: string[]; use?: {model: string; moves: number}}" optional>
  Set when the file came from `c.capture()`.
</Property>

### `JudgmentRecord`

<Property name="question" type="string"></Property>

<Property name="verdict" type="'pass' | 'fail'"></Property>

<Property name="score" type="number"></Property>

<Property name="threshold" type="number"></Property>

<Property name="model" type="string"></Property>

<Property name="resolvedModel" type="string" optional>
  The model id the provider says served the call, when it reports one.
</Property>

<Property name="gateway" type="string" optional>
  The gateway from the config's `gateway` list the call went through; unset without one.
</Property>

<Property name="upstream" type="string" optional>
  The upstream provider the gateway says served the call (OpenRouter's); unset when it names none.
</Property>

<Property name="costUsd" type="number" optional>
  What the gateway says the call cost, in US dollars (OpenRouter's `usage.cost`); unset when it names none.
</Property>

<Property name="key" type="string">
  `<index>:<sha256(question) prefix>`: the same question at the same place in the evaluation, so a verdict can be followed across runs (and an edited question starts a new series). `<index>` is the call's place among the attempt's `c.judge()` calls, counting calls that failed, whichever finishes first.
</Property>

<Property name="retries" type="JudgeRetry[]" optional>
  Transient provider errors retried before this verdict, with the backoff before each retry.
</Property>

<Property name="artifacts" type="JudgeArtifactRef[]" optional>
  Captures this call sent the judge as file parts, when any.
</Property>

### `JudgeArtifactRef`

A capture this judge call sent as a file part. Bytes live in the `artifacts/<name>.png` file.

<Property name="name" type="string"></Property>

<Property name="path" type="string">
  The `artifacts/<name>.png` path under the attempt directory.
</Property>

<Property name="sha256" type="string"></Property>

<Property name="bytes" type="number"></Property>

<Property name="mediaType" type="'image/png'"></Property>

### `JudgeRetry`

One failed judge try that was retried, and the backoff before the next.

<Property name="error" type="string"></Property>

<Property name="delayMs" type="number"></Property>

### `FileRef`

`path` is relative to the run's artifact directory; `sha256` is of the bytes on disk.

<Property name="path" type="string"></Property>

<Property name="sha256" type="string"></Property>

<Property name="bytes" type="number"></Property>

### `ContextFile`

<Property name="path" type="string"></Property>

<Property name="sha256" type="string"></Property>

### `Cell`

A cell of an evaluation combines a harness, an environment (`-` when none is set), and that environment's sandbox id. With `compare`, the control is `baseline` (or the cascaded environment, or an empty `baseline`) and each variant is that environment on its own, including its sandbox.

<Property name="key" type="string">
  The harness id, the environment name, and the sandbox id, joined with `/`.
</Property>

<Property name="harness" type="string">
  The harness id: `id` when set, otherwise `<kind>:<model>` (`<kind>` alone without a model), with `[effort=<value>]` appended whenever `effort` is set.
</Property>

<Property name="environment" type="string">
  The environment's own name, also on a variant cell (`with-skill`, with no prefix).
</Property>

<Property name="sandbox" type="string">
  The sandbox id is `id` when set, or `fake` for the fake sandbox. A Docker sandbox starts with the engine (`docker` by default). It adds `:` and the sanitized final component of `image` or `from`, or for `dockerfile` the sanitized filename (the parent directory name when that filename is `Dockerfile` and a parent is present). Non-empty `install` steps append `+` and an eight-character hash.
</Property>

<Property name="role" type="CellRole"></Property>

<Property name="variant" type="string" optional>
  The variant's name, for variant cells.
</Property>

### `EnvironmentRecord`

An environment as it was applied, without secret values.

<Property name="name" type="string"></Property>

<Property name="parent" type="string | null">
  The previous `extendEnvironment` layer's name, else null.
</Property>

<Property name="overrides" type="string[]">
  Keys a later layer replaced, for example `env.KEY`, `skills.typescript`, `mcp.docs`, or `rules`.
</Property>

<Property name="digest" type="string">
  Covers every field but the rules file's path; secret names are included; values never are. The image is the sandbox's.
</Property>

<Property name="setup" type="string[]"></Property>

<Property name="skills" type="Array<{name: string; source: string; commit: string | null; sha256: string; placement: SkillPlacement; excluded: string[]; plugin?: string}>"></Property>

<Property name="mcp" type="string[]">
  The names of the MCP servers the agent was given.
</Property>

<Property name="plugins" type="PluginRecord[]" optional>
  The plugins placed, each with what it placed; in the digest. Unset when there are none.
</Property>

<Property name="files" type="Array<{path: string; sha256: string}>"></Property>

<Property name="vars" type="string[]"></Property>

<Property name="secrets" type="string[]">
  Every secret, by the variable it's set as in the sandbox; an MCP header's (`secret.fromEnv`, `secret.template`, a plugin's `${VAR}`) by each host variable it reads.
</Property>

<Property name="brokered" type="string[]">
  Secrets the sandbox only had placeholders for, swapped in by the egress gateway, named as in `secrets`. A variable it also got as its real value somewhere (an `http://` header, an env secret without `hosts`) isn't listed.
</Property>

<Property name="unset" type="string[]">
  Optional secrets whose host variable was unset, so they weren't injected.
</Property>

<Property name="egress" type="EgressPolicy">
  The policy the attempt ran under, from the options and the environment; in the digest unless `proxy`.
</Property>

<Property name="workspace" type="{source: string; ref: string | null; path: string | null; commit: string; tree: string; init?: boolean}" optional>
  This records the Git repository the workspace started as: the repo as written (a local one relative to the project root), the ref and path asked for (null when unset), the commit the ref resolved to, and the hash of the tree that was copied in, which is in the digest.
</Property>

### `PluginRecord`

A plugin as an attempt placed it.

<Property name="name" type="string">
  The manifest's `name`.
</Property>

<Property name="source" type="string">
  Where it came from, e.g. `local:plugins/review`, `github:acme/agent-plugins#v1:plugins/review` or `npm:@acme/review@1.2.0`; for a marketplace entry, the entry's own source.
</Property>

<Property name="marketplace" type="{name: string; source: string; commit: string | null} | null">
  The marketplace it was listed in, with its `marketplace.json` name, its source as written (`github:acme/agent-plugins#v3`, `npm:@acme/plugins@2.0.0`) and the commit a git one resolved to (null for npm); null when the plugin was named directly.
</Property>

<Property name="commit" type="string | null">
  The commit a git source resolved to (the marketplace's, for an entry inside it); null for a local or npm one.
</Property>

<Property name="sha256" type="string">
  The hash of the files placed.
</Property>

<Property name="build" type="string | null">
  The tag of the image layer its build ran in (`reeval-plugin:…`); null without a build.
</Property>

<Property name="parts" type="{skills: string[]; agents: string[]; commands: string[]; hooks: string[]; mcp: string[]; rules: boolean}">
  What it placed: skill, agent, command and MCP server names, hook events, and whether it had rules.
</Property>

### `EgressRecord`

An attempt's network traffic. `allowed` is every host a restricted policy let through, its own and the automatic ones; null under `proxy` and `allow-all`. `recorded` is false when the sandbox can't see the traffic, and then `contacted` and `blocked` are empty. Bytes are counted between the sandbox and the egress gateway, TLS included. `expectedBlocked` are refused hosts the harness always tries (its CLI's telemetry, say), kept out of `blocked`, so `blocked` contains only unexpected refusals.

<Property name="policy" type="EgressPolicy"></Property>

<Property name="allowed" type="string[] | null"></Property>

<Property name="recorded" type="boolean"></Property>

<Property name="partial" type="true" optional>
  True when traffic could leave around the gateway (`allow-all` in a container that kept its network): `contacted` lists only what went through the gateway, and anything that ignored the proxy (raw TCP and UDP, SSH, database drivers) went out unrecorded. Unset otherwise.
</Property>

<Property name="contacted" type="Array<{host: string; port: number; connections: number; sent: number; received: number}>"></Property>

<Property name="blocked" type="Array<{host: string; port: number; connections: number; reason: string}>"></Property>

<Property name="expectedBlocked" type="Array<{host: string; port: number; connections: number; reason: string}>"></Property>

### `AttemptImage`

The image an attempt's container started from, and how reeval made it.

<Property name="ref" type="string"></Property>

<Property name="id" type="string">
  Content-addressed image id (`sha256:…`).
</Property>

<Property name="agentCli" type="string | null"></Property>

<Property name="base" type="string">
  The harness image the `install` layers were baked onto: `ref` itself without any.
</Property>

<Property name="install" type="string[]">
  The `docker({ install })` commands, in order.
</Property>

<Property name="dockerfile" type="{path: string; sha256: string} | null">
  `docker({ dockerfile })`: the path from the project root and the build context's hash.
</Property>

<Property name="from" type="{ref: string; digest: string | null}" optional>
  `docker({ from })`: the base as written and the digest the image was built on (a local-only base's image id); `reeval image build` rebuilds it when the tag has moved since.
</Property>

### `PhaseTimings`

The duration of each evaluation phase, in ms. `agentMs` is the harness turns; `diffMs` reading changes.

<Property name="setupMs" type="number"></Property>

<Property name="agentMs" type="number"></Property>

<Property name="diffMs" type="number"></Property>

<Property name="checksMs" type="number"></Property>

<Property name="judgeMs" type="number"></Property>

### `FileChange`

<Property name="path" type="string"></Property>

<Property name="status" type="'added' | 'modified' | 'deleted'"></Property>

### `McpStartup`

One start of a stdio MCP server during a turn (`turns[].mcpStartup`, in start order), timed by reeval's MCP timer, which every stdio server runs through. An agent CLI that starts one process per turn starts its servers on every turn; a CLI that keeps a session starts them on the first turn only. A server that doesn't answer within 60s fails the attempt as an infra error naming it; one whose client closes it first doesn't. `mcpStartup` is absent when the environment has no stdio MCP server, or the sandbox has no `node` to run the timer.

<Property name="server" type="string">
  The server's name in the environment's `mcp`.
</Property>

<Property name="ms" type="number | null">
  Milliseconds from spawning the server to its answer to the client's `initialize`; null when it never answered (it exited first, timed out, or its client closed it first).
</Property>

[//]: # (/generated:record-types)

## `RunnerInfo`

On a CI run, `RunRecord.runner` has `kind` `ci` and `provider` `github`, `gitlab`, or `unknown`. `runUrl`, `attempt`, and `pr` (`{ number, baseRef }`) are what that provider reports, or null. A local run has `kind` `local` and `hostHash`, a hash of the host name.

## `TokenUsage`

`TokenUsage` is the counts on `Cost.usage`, an attempt's `usage`, and a turn's `usage`. `input` excludes `cacheRead` and `cacheWrite`. The prompt total adds both. `reasoning` is part of `output`, or null when none are reported.

## `AttemptError`

`AttemptRecord.error` is why an attempt failed, or null. `kind` is an `ErrorKind`. `message` is the text. `prepare` is set when sandbox preparation fails. `resource` is set when the sandbox runs out of a resource.

## `PrepareFailureRecord`

`AttemptError.prepare` records a sandbox preparation that failed. `class` is `config`, `transient` after the retries, or `resource`. `stage` is `image`, `from`, `dockerfile`, `harness`, `install`, `plugin`, or `workspace`. `ref` is what was prepared, `repo#ref:path` for a Git workspace. `cause` is `enospc`, `oom`, or null. `log` is the build log or null, and `setBy` names the levels that set the sandbox.

## `ResourceFailureRecord`

`AttemptError.resource` means the sandbox ran out of memory, and the outcome is `infra_error`. `cause` is `oom`. `kills` is how many processes the kernel killed.

## `AssertionRecord`

An `AssertionRecord` is one settled `expect` or `expect.soft` call on `AttemptRecord.assertions`. `checks` on that attempt lists each `c.exec` command.

`label` is the supplied `expect` label. Without one, it is the matcher plus its first argument or named subject, cut at 60 characters. Repeats get ` #2`, ` #3`, and later suffixes. `labelled` is true when the evaluation supplies a label. `matcher` is the matcher path. `status` is `pass`, `fail`, or `error`. `soft` marks `expect.soft`. `message` is cut at 1 KB. `durationMs` is set on async matchers only.

## `StartRetry`

A `StartRetry` is one entry of `AttemptRecord.startRetries`. `error` is `<harness kind>: session did not become ready within <startTimeout>`. `waitedMs` is how long that start waited.

## `CellCheck`

`CellRecord.checks` tallies one label over the cell's counted attempts. `pass`, `fail`, `error`, and `notRun` are those counts, and `control` repeats them for the baseline. `verdict` is `discriminates` when every answered instance passes on one side and fails on the other, `both-pass` when every answered instance passes on both sides, `both-fail` when every answered instance fails on both sides, or `mixed` otherwise. It is absent when either side has no pass or fail.

## `EvalCompareMeta`

`EvalEntry.compare` names the baseline when the evaluation uses `compare`. `baseline` is that environment's name. When `compare.baseline` was omitted, `from` is `environment` for the cascaded environment, or `empty` for an empty environment named `baseline`.

## `CompareAnswer`

`EvalEntry.answers` holds one `CompareAnswer` per pairwise question. `outcome` and `passRate` do not depend on it. `question`, `model`, and `resolvedModel` identify the question and the model. `votes` holds `PairVote` values, and `rates` holds `wins` out of `n` per `cut` and `name`. `error` is set when the model call fails. `artifacts` is `true` or capture names. `input` is a `FileRef` for the `questions/<n>.json` file. `stateOptions` is the state each side showed.

For more information about `stateOptions`, see [the `question` option](/reference/options#question).

## `PairVote`

A `PairVote` judges one pair in one order. `cut` is `within` (two environments, one harness) or `across` (two harnesses, one environment). `harness` is set on `within`, and `arm` names the environment on `across`. `run` starts at 1. `a` and `b` name the sides, and `winner` is `a`, `b`, or `tie`. `swapped` is true when `b` was shown first. `reason` and `artifacts` are omitted when absent.

## `AttemptRecord.turns`

A turn's `usage` and `model` are null when the harness reports neither. `error` is the first error, cut at 1 KB, or null. `unreported` lists `web-search` or `web-fetch` when that event cannot be reported.

## `AttemptRecord.activity`

`webSearches` and `webFetches` are null when a turn cannot report them. `stalledTools` counts stalled tool events, and `maxToolMs` is the longest tool duration, or null.