--- title: reeval run description: Plan and run evals, with every flag, filter rule and output option. --- > ## Documentation Index > Fetch the complete documentation index at: https://docs.reeval.ai/llms.txt > Use this file to discover all available pages before exploring further. `reeval run` plans the evals that match `include` and your filters, prints the plan, runs every attempt and prints a result block per eval. ```bash npx reeval run hello --runs 2 --ui plain ``` ## Help [//]: # (generated:cli-help:run) ```text reeval run [filter...] Run evals matching config.include; filters are path, name or cell substrings (parts of one kind OR; a name and a cell key are one kind) --profile Size the run with a profile from reeval.config.ts (also REEVAL_PROFILE; default: defaultProfile.ci in CI, else defaultProfile.local) --dry-run Print the plan (evals, cells, attempts, time and cost) and exit 0; with --json, as one JSON document (every cell, and what's excluded) --json Print the run's results (suiteResult from reeval/view) as one JSON document on stdout when it ends; progress goes to stderr --force Run even when the plan is over budget.attempts --runs Attempts per cell, overriding the config, groups and evals --concurrency Attempts running at once across evals and cells (default 8, at most one per CPU and GB of Docker memory; also REEVAL_CONCURRENCY) --tag Only evals with this tag; !tag excludes it; repeat to combine (AND) --record Also record the run into the history folder (see config.record) --ui live, plain or agent (default: agent under a coding agent, plain in CI or when piped, else live; live prints plain for now) --log-order stream (every line as it happens) or grouped (each attempt as one block when it ends); default grouped in CI, else stream --log-prefix task or none: the [eval#n harness] prefix on detail lines (default task, none inside GitHub Actions groups) --output-logs full, errors-only or none: whose detail lines print (default errors-only, failing attempts only; full with -v) -v, --verbose Print each attempt's phase trace as it happens: plan, image, container, environment, brokering, turns and tool calls, checks, judge calls, gateway requests and teardown -vv, --trace Also raw payloads, docker commands, headers and DNS decisions, and keep every attempt's trace.ndjson (else only failing attempts') Either way each attempt's debug.log is in .reeval/debug// (local only) --resume [ref] Continue a stopped, interrupted or budget-stopped run: same run id, skip completed attempts, restart in-flight ones. Without a ref, the newest resumable (Stopped staging, or an interrupted/budget scratch), not assembled latest. Ref is a run id or unique prefix --no-insights Skip insights for this run: no analysis.model calls, the blocks print their facts only (also analysis.insights: false) ``` [//]: # (/generated:cli-help:run) ## Filters ```bash npx reeval run evals/hello.eval.ts # a file npx reeval run "greet module" # evals by group or name npx reeval run "greet module › creates" # an eval in a group, by its id npx reeval run with-house-greeting # cells by key npx reeval run skill antigravity # skill.eval.ts's Antigravity cells ``` - A filter part that names an eval file picks files. Every other part picks evals by name or cell key, as a case-sensitive substring of the eval id or the cell key (`cursor:composer-2.5/with-house-greeting/docker`). - Parts of one kind add up (OR). The two kinds narrow each other, and the plan's rows and totals count only the cells that are left. - An eval name and a cell key, harness included, are the same kind. `reeval run claude-code:claude-sonnet-5 "greets by name"` plans every Claude Code Sonnet cell plus `greets by name` on every harness. To run one eval on one harness, filter by the eval and set a profile's [`limit.harness`](/reference/profiles#limit). - A part that matches nothing exits 2 and names it. - `--tag slow` keeps evals tagged `slow`, `--tag '!slow'` drops them. Repeated `--tag`s must all hold. A profile's `tags` work the same way. ## The plan line Every run starts with it, before anything runs: ```text plan run 20260101T000000Z_nogit_ada-lovelace_k3f9 · 1 eval · 2 cells · 4 attempts · ~50s serial (~13s at concurrency 4) ``` The estimate is the median of each eval's recent attempts on that harness (the last 30 runs, recorded and scratch), else the harness's median, else 3 minutes an attempt. A `fakeHarness` attempt with no history is 1 second in `fakeSandbox()` and 30 seconds in `docker()` or `podman()`, the sandbox's setup. With a profile and a budget it also says which profile, why, and whether the plan fits: ```text plan run 20260926T174055Z_b010670_ci_vdfy · profile pr (auto: CI is set) · 18 evals · 19 cells · 19 attempts · ~57m 00s serial (~14m 15s at concurrency 4) · budget 20 attempts ✓ ``` `--dry-run` adds a row per eval, then, when Docker is running, a line per sandbox image saying whether it's ready, to build, stale or missing (a `from` tag is looked up in its registry, never pulled), and stops with exit 0. Neither needs credentials, and Docker is optional. `--dry-run --json` prints the whole plan, every cell and what was excluded, as one JSON document, with each image step's `status` and `detail`. ```text images 4 sandbox images: 1 ready, 3 to build ✓ docker · cursor ready + docker+0ab0e263 · cursor + install to build: install 1 command, not built yet + python · cursor to build: from python:3.12-slim, on python:3.12-slim (f77ac9e44ae9) … ``` ## The cost estimate ```text cost estimated $1.20–9.00 at prices.json over models.dev list prices, judge included · no history, rough cost $0.77–6.12 priced (claude-sonnet-5, gpt-5.6-luna, judge) at models.dev list prices; composer-2.5: 1–13M tokens, no list price · no history, rough cost estimated 5–38M tokens, judge included · no prices: models.dev unreachable, next try in 60m · no history, rough ``` `--dry-run` ends its plan with what the run should cost: tokens, priced in USD where it has a price for the model. It's an estimate for deciding whether to run, not your bill: subscriptions and included usage, negotiated prices and re-runs aren't in it. | Part | Where its tokens come from | |---|---| | Agent | The same eval and cell's counted attempts in the last 30 runs (recorded and scratch), else the same eval on that harness: their middle half, never narrower than −20% / +25% of the median. With neither, a typical coding attempt: 300k–2M prompt tokens, mostly cached, and 3–20k out | | Judge | As many calls an attempt as the eval's past attempts made, on their model, else 0–1 calls on the run's `judge.model`; 5–60k tokens in and 0.5–2k out a call | When some models have no price, the line doesn't add up to a total: it says what the dollars cover (`priced (judge only)`, `priced (claude-sonnet-5, judge)`) and lists each unpriced model's tokens. Reasoning tokens are priced as the output they're part of. Without history, judge calls are priced on the run's `judge.model`, even for an eval or group that sets its own. Once the eval has run, its past judge calls set the model. reeval keeps no prices of its own, and never writes one to a record. The dry run looks them up as it prints: - **[models.dev](https://models.dev)**, by default: the Anthropic, OpenAI, Google and xAI list prices, and Vercel AI Gateway's `provider/model` ids, cached for a day in reeval's host cache directory. The request sends nothing about your project, gives up after 2 seconds or 32 MB, and goes through `HTTPS_PROXY` when `NODE_USE_ENV_PROXY=1` is set (Node 22.21 or later), as the judge's calls do; a failure isn't retried for an hour. Offline, or when models.dev answers without prices, it uses an older copy and says so (`models.dev list prices of 2026-10-20 (offline)`, `(models.dev answered HTTP 500)`); with none, the line is tokens only. - **Your own**, with `REEVAL_PRICES=prices.json`: USD per million tokens by model id, in models.dev's field names (`input`, `output`, `cache_read`, `cache_write`), over models.dev's. A relative path is from the working directory. Any other field, ids that differ only in case, or a file that isn't such an object stops the dry run with exit 2. A model in the file takes its price from there, and every other from models.dev. Cursor's models aren't on models.dev, so this is how they get a price. When models.dev can't be reached, the file's prices are used alone. `REEVAL_PRICES=none` is tokens only, and fetches nothing. - **Model ids** match as written (`anthropic/claude-sonnet-4.5`) or without the `provider/` prefix, case-insensitively and without a trailing effort (`gemini-3.8-flash-medium`), and Cursor's `params: { fast: 'true' }` looks up `composer-2.5-fast`. The fake agent and judge cost nothing, and questions and insights, cents, aren't counted. ```json title="prices.json" { "composer-2.5": { "input": 0.5, "cache_read": 0.2, "output": 2.5 } } ``` `--dry-run --json` has the detail under `cost`: `tokens` and `usd` as `{ low, high }` (`usd` null without prices), `agent` and `judge` each with `tokens` and `usd`, `unpricedTokens`, `attempts` by basis (`history`, `typical`, `free`), `pastRuns`, `pastAttempts`, `unpriced` (each model with no price and its `tokens`), `prices` (`modelsDev`, with `fetchedAt`, `stale` and, when stale, `why` the fetch failed, or only `why` there's no copy; and the `REEVAL_PRICES` `file`; or `why` there are no prices), and `cells`, each with its `model`, `judgeModel`, `basis`, `samples`, `attempts`, `agent` and `judge`. ## Results as JSON ```bash npx reeval run --config reeval.fake.config.ts --json > results.json ``` ```json { "runId": "20261021T064035Z_02e3770_cursor-agent_nwqx", "status": "complete", "outcome": "pass", "errors": [], "tally": { "errors": 0, "pass": 16, "notGated": 0, "fail": 0, "infraError": 0, "cancelled": 0, "skipped": 0 }, "files": [ …, { "path": "evals/skill.eval.ts", "evals": [ { "path": "evals%2Fskill.eval.ts › house greeting › greets in the house style", "name": "greets in the house style", "across": "change", "family": false, "outcome": "pass", "cells": [ { "key": "fake/plain/fake", "baseline": true, "gated": false, "passed": 0, "attempts": 1, "checks": [ { "label": "house greeting", "outcome": { "status": "ran", "passed": 0, "failed": 1, "errored": 0, "notRun": 0, "assertion": "house greeting: expected command to succeed, …" }, … } ], … }, … ] } ] } ], "paths": ["evals%2Fgreetings.eval.ts › greets %25s in the house style", …] } ``` `--json` prints the run's facts as one JSON document on stdout when the run ends; the progress and result blocks go to stderr. It's `suiteResult(record)` from `{{ pkg }}/view`, recomputed from the record, so `reeval show --json` still prints the record itself. The exit code is the same as without `--json`. | Field | What | |---|---| | `outcome` | The run's outcome, which the exit code is read from: `skipped` when interrupted, else `fail` on any error outside an attempt | | `errors` | Errors outside any attempt: an eval file's stray rejection or throw, a worker that died. Each fails the run | | `tally` | Outcomes per block: each `evaluate.each` case and each other eval is one, and a family whose pooled `minLift` failed is one more fail. `notGated` counts the studies (every cell `gate: 'none'`), which `pass` leaves out, `cancelled` the blocks whose only failed tries the interrupt cancelled, and `errors` the run's `errors` | | `files[].evals[]` | An eval, or a pooled `evaluate.each` (`family: true`) with its `cases`, `pooled` variants and `questions` summed over the cases | | `across` | What the cells compare: `none`, `change` (variants), `harness` or `both` | | `cells[]` | Each cell: counts (`infraError` and `skipped` apart from `failed`), `baseline`, `gated`, `fingerprint`, `context` (skill sources and files) | | `checks[].outcome` | `ran` with its counts and first failure, or `not run` when no counted attempt reached it. A cell with no counted attempt lists every check the eval (or family) recorded as `not run` | | `effect` | A variant against its baseline: `better`, `worse`, `same` or `unclear`. Fewer than 4 counted attempts a side is `unclear`, and so is a check the baseline never ran | | `effort` | Median tool calls, agent time and tokens, from 4 counted attempts | | `paths` | Every node's `path`, a stable id to cite it by: `//++/checks/