---
title: Options
description: >-
Every option groups, evals, profiles and the config accept, and how each one
inherits.
---
> ## Documentation Index
> Fetch the complete documentation index at: https://docs.reeval.ai/llms.txt
> Use this file to discover all available pages before exploring further.
`defineConfig`, `group` and `evaluate` take the same options. They cascade config → profile `defaults` → outer group → inner group → eval, then the profile's `limit` narrows them, then `--runs` overrides.
```ts title="evals/skill.eval.ts"
group(
'house greeting',
{
compare: { baseline: plain, variants: [withHouseGreeting] },
runs: 2,
pin: ['compare'],
},
() => {
evaluate(
'greets in the house style',
{ timeout: '10m', tags: ['skills'] },
async (c) => {
// …
},
)
},
)
```
## Where each key is accepted
[//]: # (generated:option-keys)
| Key | `reeval.config.ts` | Profile `defaults` | Group or eval | Profile `limit` |
|---|---|---|---|---|
| [`include`](#include) | ✓ | | | |
| [`gateway`](#gateway) | ✓ | | | |
| [`harness`](#harness) | ✓ | ✓ | ✓ | ✓ |
| [`sandbox`](#sandbox) | ✓ | ✓ | ✓ | |
| [`environment`](#environment) | ✓ | ✓ | ✓ | |
| [`runs`](#runs) | ✓ | ✓ | ✓ | ✓ |
| [`gate`](#gate) | ✓ | ✓ | ✓ | |
| [`env`](#env) | ✓ | ✓ | ✓ | |
| [`egress`](#egress) | ✓ | ✓ | ✓ | |
| [`judge`](#judge) | ✓ | ✓ | ✓ | |
| [`capture`](#capture) | ✓ | ✓ | ✓ | |
| [`question`](#question) | ✓ | ✓ | ✓ | |
| [`timeout`](#timeout) | ✓ | ✓ | ✓ | |
| [`concurrency`](#concurrency) | ✓ | | | |
| [`record`](#record) | ✓ | | | |
| [`history`](#history) | ✓ | | | |
| [`budget`](#budget) | ✓ | | | |
| [`profiles`](#profiles) | ✓ | | | |
| [`defaultProfile`](#defaultprofile) | ✓ | | | |
| [`secrets`](#secrets) | ✓ | | | |
| [`debug`](#debug) | ✓ | | | |
| [`analysis`](#analysis) | ✓ | | | |
| [`compare`](#compare) | | | ✓ | ✓ |
| [`questions`](#questions) | | | ✓ | |
| [`tags`](#tags) | | | ✓ | |
| [`pin`](#pin) | | | ✓ | |
[//]: # (/generated:option-keys)
Config-only keys are on [defineConfig](/reference/config); `profiles` and `defaultProfile` are on [Profiles and budgets](/reference/profiles). Options are plain objects: a function in the options position is an error, and an unknown key is exit 2 with a hint (`Unknown option "repeat" in evaluate("e"). Did you mean "runs"?`).
## Options
Which agents run the eval. **Replaces** the inherited value. The only list: each harness is its own cell. An edit changes the inherited list instead: `{ add: cursor({ model: 'composer-2' }) }` appends one or a list, `{ only: 'cursor:composer-2.5' }` keeps the inherited harnesses with these ids (one or a list), and a function gets the list and returns a new one, `(h) => [...h, cursor({ model: 'composer-2' })]`. An empty list, two harnesses with the same id, or an `only` id nothing inherited has, is an error. Default `cursor({ model: 'composer-2.5' })`.
The default machine when the resolved environment omits `sandbox`. **Replaces**. Cascades config → profile `defaults` → group → eval. Not an environment: it does not invent skills, files or a name. A named environment is a total override and does not inherit the config environment's sandbox. `.extend()` keeps the parent environment's sandbox when the child omits it. Default `docker()`.
What the eval runs in, from `defineEnvironment`. **Replaces**. One environment, never a list. Used only when the eval or compare arm did not name an environment; a named env is a total override (no skills, files or sandbox from this one). Beside or above a compare without a `baseline`, it's the control — including a default `environment` on the config. Default none: the attempt uses the cascaded `sandbox` option, else implicit Docker.
A control and other environments, each run on its own and reported as lift. **Replaces**. Groups and evals only: it's refused in `reeval.config.ts`. `compare: false` is not a value here. A profile turns compare off with `limit.compare: false`, which runs the control alone. `compare` does not layer.
- `baseline`: the control, an environment from `defineEnvironment`. With it, `environment` can't sit on the same object, and an inherited one is not applied as files, skills or variables. A named arm that omits `sandbox` uses the cascaded `sandbox` option, else implicit Docker; it does not inherit the config environment's sandbox. `.extend()` keeps the parent environment's sandbox when the child omits it. Without a `baseline`, the control is the cascaded `environment` (the same object's, a group's or the config's), else an empty environment named `baseline`. That empty control uses the first variant's sandbox when the variant sets one, so the compare is the same machine without the variant's files, not implicit Docker versus Node 24. A variant with the control's name is an error. The record's `evals[].compare` names the control and, for an omitted baseline, sets `from` to `environment` or `empty`.
- `variants`: the other environments. Each cell runs that environment alone.
- `gate`: `'variants'` (default) gates only variant cells; `'all'` gates the control too; `'none'` only reports.
- `minLift`: a number from -1 to 1. A variant cell fails when its lift interval's lower bound is below it. On a pooled `evaluate.each`, it gates the family's paired lift over the cases, not each case's lift.
- `harness`: harness ids to compare on; the others run the control only. An id the eval doesn't run is an error.
Attempts per cell, a whole number of at least 1. **Replaces**. `reeval run --runs n` overrides every level, pins included. Default 1.
How a cell passes. **Replaces**. `'all'` (default) is `{ passRate: 1 }`: every counted attempt must pass. `{ passRate: 0.8 }` passes when at least 80% of passes and fails are passes. Infra errors don't count; a cell is `infra_error` only when nothing counted.
Variables for the sandbox. **Merges by variable name**. Secrets come from a `defineConfig` or `defineEnvironment` callback's `secret` helper; a literal string under a secret-looking name is refused.
Where the sandbox may connect. **Replaces**. An environment's `egress` sits between the profile `defaults` and the groups. Default `'proxy'`. See [Egress](/reference/egress).
Judge settings. **Merges field by field**, so overriding `threshold` keeps the model. No default model; `threshold` defaults to 0.8. See [Judges](/reference/judges).
Questions a model answers about the eval's attempts. **Replaces**: an eval's list replaces its group's. Groups and evals only: refused in `reeval.config.ts` and not a profile default. Questions annotate and never change pass, fail or the exit code. See [Questions](/guide/questions).
- `question`: the question. A plain string is an entry with no other keys.
- `across`: `'environment'`, `'harness'`, or a list of those. Set, the question is pairwise: after every attempt of the eval ends, the attempts that passed are paired by varying those factors and holding the others, and each pair is voted on in both orders. Unset, the question is asked of each attempt as it ends with a pass or a fail (four calls at a time; infra errors, skipped and cancelled attempts aren't asked). A factor the eval's matrix doesn't vary exits 2; one a profile's limit narrows to a single level drops the question with a note.
- `choices`: two or more distinct, non-empty answers the model picks one of, with a short reason. Per-attempt questions only: with `across` it exits 2. Without it, the answer is free text.
- `model`: a `'provider/model'` language model. Without one, the cascaded [`question.model`](#question) answers, else `analysis.model`, else the run's judge model (a profile's over the config's) when it's a language model. `reeval run` exits 2 at plan time when a question has no language model or would get an evaluation model.
- `state` and `artifacts`: what the model is shown, over the cascaded [`question`](#question) option.
`--dry-run` names each question, what it shows, and how many attempts or pairs it will ask (assuming every attempt passes its checks). An empty list asks none, so an eval can clear its group's. An empty string, a value that isn't a list, an unknown entry key and the same question twice are errors. Per-attempt answers land in `evals[].attempts[].answers` and `reeval query attempt_answers`; pairwise ones in `evals[].answers`, `votes` and `question_pairs`.
The model that answers a question, and what it shows. **Merges**: `state` key by key, `model` and `artifacts` replaced. Cascades config → profile `defaults` → group → eval → the [`questions`](#questions) entry. `judge.state` never reaches a question.
- `model`: the `'provider/model'` language model that answers every question below it that names no `model` of its own. An entry's `model` wins; without either, `analysis.model` answers, else the run's judge model (a profile's over the config's) when it's a language model. An evaluation model or a string without a provider exits 2. Its key is checked before the run like the entry's.
- `state`: the judge's [`state`](/reference/judges#state) keys, per attempt or side, except `contents` (a question runs after its attempts' workspaces are gone). The defaults differ from the judge's. A per-attempt question shows `task`, `output`, `diff` and `transcript` (`files: false`, `turns: 'all'`, `checks: false`). A pairwise one shows `task: true`, `output: false`, `files: false`, `diff: true`, `transcript: false`, `turns: 'all'`, `checks: false`: side by side, the more confident summary would win on wording. A task both sides were given goes in the prompt once; different prompts go in each side. `checks` names a command, and an attempt that didn't run it shows it as not run.
- `artifacts`: `true` for every capture each attempt took, in capture order, or a non-empty list of capture names. Each attempt sends its own capture of that name as a PNG file part; in a pair, each side's is labelled only by its position (`Attempt A, screenshot 1 of 1`). A name an attempt lacks is `not captured` there. Default none.
A question that shows nothing (every key off and no `artifacts`) exits 2. A pairwise one that shows `transcript` and pairs two harnesses gets a plan warning: the tool names say which agent each side is. `across` and `choices` are per entry, not here. The answer records `stateOptions` and `artifacts`, and each vote the captures each named side sent. A name or command no paired attempt has fails that answer with `error`.
Camera defaults for `c.capture()`. **Shallow-merged**; a level that sets `viewport` replaces the whole value. `viewport` is `{ width, height }` in CSS pixels (default 1280×720) or `'full-page'`. `timeout` is how long to poll before failing (default `'60s'`). `browser: false` refuses the host `chrome-headless-shell` binary: `c.capture()` then throws. `model` is the vision model; it cascades like `question`. Without it, `analysis.model` drives the journey, else the attempt's judge model (a profile's, group's or eval's over the config's), whichever is first a language model; never an evaluation model such as Jev. A Gemini model's points come back normalised to 0–1000, and reeval scales them to the screenshot. `wait` and `instructions` are call-only. `port` is not an option here: pass it as the first argument to `c.capture`. See [The attempt context](/reference/attempt-context) and [Capturing UI](/recipes/capturing-ui).
Per attempt: milliseconds, or `'500ms'`, `'90s'`, `'15m'`, `'1h'`. **Replaces**. Image builds and setup steps don't count, and the clock starts when the attempt gets a concurrency slot. Default `'15m'`.
Labels for `reeval run --tag`. **Union** of every level.
Keys a profile's `limit` may not narrow here. **Union**. Anything else is an error.
## Precedence, top wins
| Level | Where |
|---|---|
| `--runs` | The command line (runs only) |
| Profile `limit` | Narrows only, unless the key is pinned |
| Eval | `evaluate(name, options, fn)` |
| Inner group, then outer group | `group(name, options, fn)` |
| Environment `egress` | For `egress` only |
| Profile `defaults` | Fills in what nothing else set |
| Config | `reeval.config.ts` |
## Related
- [Options and inheritance](/concepts/options)
- [defineConfig](/reference/config)
- [Profiles and budgets](/reference/profiles)