---
title: defineConfig
description: >-
Every key of reeval.config.ts and reeval.yaml, its default and type, the
callback's helpers, and the errors reeval raises for a bad config.
---
> ## Documentation Index
> Fetch the complete documentation index at: https://docs.reeval.ai/llms.txt
> Use this file to discover all available pages before exploring further.
`reeval.config.ts` default-exports `defineConfig(…)`: the project's defaults, profiles and run settings.
```ts title="reeval.config.ts"
import { defineConfig } from '{{ pkg }}'
import { cursor } from '{{ pkg }}/harness'
export default defineConfig(({ ci, secret }) => ({
harness: cursor({ model: 'composer-2.5' }),
judge: { model: 'typesafe-ai/jev-1.14.0' },
env: { NPM_TOKEN: secret({ hosts: ['registry.npmjs.org'] }) },
concurrency: ci ? 2 : 4,
}))
```
An omitted key takes its default. The option keys (`harness`, `sandbox`, `environment`, `runs`, `gate`, `env`, `egress`, `judge`, `capture`, `timeout`) are the start of the cascade described on [Options](/reference/options); this page covers what's specific to the config. `sandbox` is the default machine when the resolved environment omits one; it is not an environment.
## Loading
```text
my-project/
reeval.config.ts ← found from my-project/ and every directory below it
.env.local
evals/
packages/app/
reeval.config.mjs ← nearer: packages/app/ and below use this one
src/lib/
```
Without `--config`, `reeval` looks in the current directory, then in each parent up to the filesystem root, and uses the first config it finds, as Vitest and ESLint do. In each directory it tries `reeval.config.ts`, `.mts`, `.cts`, `.js`, `.mjs` and `.cjs`, in that order, and then [`reeval.yaml`](#reevalyaml) or `reeval.yml`. A directory with a YAML config and another config is an error.
A git or workspace root doesn't stop the search, so a nested project with its own config uses it, and a project without one uses its parent's.
| You run | reeval reads |
|---|---|
| `reeval run` in `my-project/evals` | `my-project/reeval.config.ts` |
| `reeval run` in `packages/app/src/lib` | `packages/app/reeval.config.mjs` |
| `reeval run --config ../reeval.claude.config.ts` | that file, relative to the current directory |
| `reeval run --config reeval.staging.yml` | that YAML config |
`--config` always wins, and names like `reeval.claude.config.ts` are only read through it. A missing `--config` file is an error (`Config file not found: nope.ts`).
`reeval run` and `reeval image build` also fail when no config is found (`No reeval.config.ts … here or above. Run reeval init.`). Other commands then use each key's default in the current directory.
The config's directory is the project root, wherever you run `reeval`. `include` globs, relative paths in the config (see [Relative paths](#relative-paths)), `.env.local` and `.env`, `.reeval/` (scratch runs, debug data, `history.db`), the history folder and `reeval studio build`'s default `studio-dist/` all resolve against it. So every command behaves the same from a subdirectory as from the root.
Env file precedence: the shell, then `.env.local`, then `.env`.
```text title="reeval run -v, in my-project/src/lib"
plan config ../../reeval.config.ts · 1 eval · 1 cell · 1 attempt · ~1s serial
…
[greets#1 fake] 10:28:24 plan config ../../reeval.config.ts · no profile · …
```
The plan line names the config when it isn't in the current directory, and `-v` names it on each attempt's first `plan` line.
The config is TypeScript loaded as is, with no build step. It must default-export an object or a callback that returns one: anything else is `reeval.config.ts's default export must be defineConfig({…}) (got …)`.
## The callback
Pass a function to get helpers and facts about the run. It runs once, when the config loads, and must return the config synchronously: a `Promise` is refused.
Marks a value as a secret read from the host: `secret()`, `secret.fromEnv('HOST_VAR')`, `secret.template(…)`, `secret.hosts.*`. See [Helpers](/reference/helpers#secret).
Skill sources: `skill.local`, `skill.git`, `skill.npm`, `skill.registry`, `skill.inline`. Local paths resolve against the config's directory. See [Helpers](/reference/helpers#skill).
MCP servers: `mcp.stdio`, `mcp.http`, `mcp.sse`. See [Helpers](/reference/helpers#mcp).
A host file, relative to the config's directory, read when an attempt starts. See [Helpers](/reference/helpers#file).
`true` when `CI` is set to anything but `false`.
The command loading the config: `run` (also for `reeval image build`), `studio`, `mcp`, `clean`, or a history command (`record`, `history`, `show`, `compare`, `checkout`, `query`, `analyze`).
`process.env` of the `reeval` process, after `.env.local` and `.env` are loaded.
The profile named by `--profile` or `REEVAL_PROFILE`. It's `undefined` when the profile comes from `defaultProfile`, which is resolved after the callback returns.
`defineConfig` itself does nothing at runtime: it returns what you pass, and exists for types.
## Relative paths
```ts title="reeval.config.ts"
sandbox: fakeSandbox({ dir: './sandboxes' }), // or { kind: 'fake', dir: './sandboxes' }
```
```yaml title="reeval.yaml"
sandbox: { kind: fake, dir: ./sandboxes }
```
Every host path follows one rule: a relative path resolves against the directory of the file it's written in, wherever you run `reeval` from. Absolute paths stay as written. A helper and its data form resolve alike, so all three lines give `sandboxes/` beside the config.
In a config, that's the config's directory, for `reeval.config.ts` and `reeval.yaml` alike. In an eval file, or a module that calls `defineEnvironment` or a helper, it's that file's. An eval file's declarations sent as data, by an SDK in another language, resolve against that eval file's directory too.
| Path | Helper | Data form |
|---|---|---|
| A host file in `files` or `rules` | `file(path)` | `{ kind: file, path }` |
| A local skill | `skill.local(path)` | `{ kind: skill, source: local, path, placement }` |
| A local npm skill (`./`, `../`, absolute or `file:`) | `skill.npm(spec)` | `{ kind: skill, source: npm, spec, path, placement }` |
| A local git workspace | `git(path)` | `{ kind: git, repo, url }`, with `url` resolved |
| A Dockerfile, which must be a file | `docker({ dockerfile })` | `{ kind: docker, dockerfile }` |
| A fake sandbox's parent directory | `fakeSandbox({ dir })` | `{ kind: fake, dir }` |
| A host directory's files | `directory(path)` | each file as `{ kind: file, path }` |
## reeval.yaml
```yaml title="reeval.yaml"
harness: { kind: cursor, model: composer-2.5 }
judge: { model: typesafe-ai/jev-1.14.0 }
env:
NPM_TOKEN: { kind: secret, hosts: [registry.npmjs.org] }
environment:
kind: environment
name: base
layers: [{ name: base, spec: { setup: [npm ci] } }]
defaultProfile: { ci: pr }
profiles:
pr: { limit: { runs: 1, compare: false } }
```
A project without TypeScript can write its config as YAML. It takes the same keys, each in its data form: what the helpers return, written out. Eval files stay TypeScript.
| In `reeval.config.ts` | In `reeval.yaml` |
|---|---|
| `cursor({ model: 'composer-2.5' })` | `{ kind: cursor, model: composer-2.5 }`, and so on for each harness |
| `docker({ from: 'node:24' })`, `podman({ from: 'node:24' })`, `fakeSandbox()` | `{ kind: docker, from: 'node:24' }`, `{ kind: docker, engine: podman, from: 'node:24' }`, `{ kind: fake }` |
| `secret({ hosts: [...] })`, `secret.fromEnv('X')` | `{ kind: secret, hosts: [...] }`, `{ kind: secret, from: X }` |
| `defineEnvironment('base', spec)` | `{ kind: environment, name: base, layers: [{ name: base, spec }] }` |
| `file('./notes.md')` | `{ kind: file, path: ./notes.md }` |
| `skill.local('./skills')` | `{ kind: skill, source: local, path: ./skills, placement: harness }` |
| `git('../app')` | `{ kind: git, repo: ../app, url: ../app }` |
| `gateway: [vercel(), direct()]` | `gateway: [vercel, direct]` |
| `fakeJudge({ score: 0.9 })` | `{ kind: fake-judge, score: 0.9 }` |
| `fakeAnalysis({ note, vote })` | `{ kind: fake-analysis, note, vote }`, with fixed values |
There is no callback, so `ci`, `hostEnv` and the helpers aren't available, and AI SDK model objects and score functions stay TypeScript only. Its shape is a JSON Schema, `packages/reeval/schema/plan/config.schema.json` in reeval's repository, which a YAML language server can check the file against.
Relative paths in it follow [the same rule](#relative-paths) as in `reeval.config.ts`.
### Mistakes in reeval.yaml
An empty file is an empty config. A YAML syntax error names the line and column. Every key gets the checks `reeval.config.ts` gets, and the config contract then refuses anything else that isn't its data form, naming the field:
```text
reeval: Unknown option "dirr" in history in reeval.yaml. Did you mean "dir"?
reeval: judge.model.score in reeval.yaml must be at most 1 (got 2)
```
## Keys
Globs of eval files, relative to the config's directory. Default `['evals/**/*.eval.ts']`. Must be a list of strings.
The agents every eval runs on, unless a group or eval replaces them. Default `cursor({ model: 'composer-2.5' })`, whose id is `cursor:composer-2.5`. An empty list, or two harnesses with the same id, is an error. See [Options](/reference/options#harness).
The default machine when the resolved environment omits `sandbox`. Not an environment: it does not invent skills, files or a name. Profile defaults, groups and evals cascade over it. Default `docker()`. See [Options](/reference/options#sandbox).
The environment every eval runs in when the eval or compare arm did not name one, from `defineEnvironment`. This is the control when a compare omits `baseline`. A named environment is a total override: no skills, files or sandbox inherit from this one. Default none. See [defineEnvironment](/reference/environment).
Attempts per cell. A whole number of at least 1. Default `1`.
What makes a cell pass: `'all'` (every counted attempt passes) or `{ passRate }`, a share above 0 and at most 1. Default `'all'`.
Variables for the sandbox. A literal string under a secret-looking name (containing `KEY`, `TOKEN`, `SECRET`, `PASSWORD` or `CREDENTIAL`) is refused. A required secret that isn't set stops the run before it starts. Default `{}`.
Which hosts the sandbox can reach. Default `'proxy'`. See [Egress](/reference/egress).
Judge settings. In the config, `model` must be a `"provider/model"` string or `fakeJudge()`; pass model objects per group, eval or call. Default threshold `0.8`, no model. See [Judges](/reference/judges).
How long one attempt may take: `'90s'`, `'15m'`, `'1h'`, `'500ms'`, or a number of milliseconds, more than zero. Default `'15m'`. Time spent starting the sandbox and running setup doesn't count.
The gateways model calls may go through, in the order you prefer them: each harness, the judge and analysis take the first that can serve them, unless they pin one with their own `gateway`. Default `[direct()]`, each provider's own key. An empty list, an unknown gateway or one listed twice is an error. See [Gateways](/reference/gateway).
How many attempts run at once. A whole number of at least 1. Default: the smallest of 8, the CPU count, and the memory in GB, measured on the Docker VM (or the host when Docker isn't available). `--concurrency` beats `REEVAL_CONCURRENCY`, which beats this key.
When `reeval run` writes a run to history. `ci` records only when `CI` is set; `manual` leaves it to `reeval record`. Default `'ci'`. A run stopped by `budget.duration` is recorded too, with status `budget`, and `reeval compare` never takes it as a base; an interrupted one never is. With `manual`, `reeval record` needs `--allow-incomplete` for it.
`dir` is the history folder, relative to the config's directory; default `.history` inside the folder of your eval files (`evals/.history`). `artifacts` says which attempts keep their transcripts, diffs and logs; default `'all'`.
Caps a run: `attempts` refuses a plan with more attempts, `duration`, counted from the first attempt's start, lets running attempts finish and skips queued ones once it's reached. A profile's `budget` is merged over this one. See [Profiles and budgets](/reference/profiles#budget).
Named presets of `defaults`, `limit`, `judge`, `budget`, `record` and `tags`. See [Profiles and budgets](/reference/profiles).
The profile used when neither `--profile` nor `REEVAL_PROFILE` names one: `ci` when `CI` is set, `local` otherwise. Each must name a key of `profiles`.
What to do about a secret the sandbox gets as a real value (one with no `hosts`). `warn` prints a line in the plan; `error` stops the run. Default `'warn'`. See [Credential brokering](/concepts/credential-brokering).
`keep` is how many runs of `.reeval/debug` data to keep (default `20`); `traceBufferMb` caps each attempt's in-memory trace (default `8`). See [Debug data](/reference/debug-data).
Settings for `reeval analyze` and the run's insights. `model` is a `provider/model` language model (default: the judge's, when it's a language model, except for insights, which use `analysis.model` only); `instructions` add your own guidance, placed first in the prompt as the question the analysis answers; `maxInputTokens` caps `reeval analyze`'s digest (default `20000`, at least 1000) and `maxOutputTokens` its answer (default `8000`, at least 500); insights have their own fixed caps. `model` also answers a [compare `question`](/reference/options#compare) that names no model of its own. `anthropic/*`, `google/*` and `openai/*` models call their provider directly with `ANTHROPIC_API_KEY`, `GOOGLE_GENERATIVE_AI_API_KEY` or `OPENAI_API_KEY`; any other prefix needs `vercel()` in [`gateway`](/reference/gateway), which takes `AI_GATEWAY_API_KEY` or `VERCEL_OIDC_TOKEN`, or `openrouter()`, which takes `OPENROUTER_API_KEY` ([Providers and keys](/reference/judges#providers-and-keys)). Without a key that works, the plan warns that insights will be skipped. See [reeval analyze](/reference/cli/analyze).
With `model` set, `reeval run` writes [insights](/reference/cli/analyze#insights) once its attempts finish: `insights: false` (default `true`) or `--no-insights` turns them off. `maxTokens` (default `250000`) caps the tokens, in and out, of every insight call in one run. It's spent on failing evals first, then on changes that made something worse, then on compares, each eval reserving an estimate before it starts, so no call starts once the budget is spent; the evals it doesn't reach print their facts, and the run prints `insights skipped for N evals`. Insights run to the end with no overall time limit, within fixed caps per eval and per call: [the caps](/reference/cli/analyze#the-caps). Ctrl-C while they run skips the rest and keeps the run's exit code. Insights never change the exit code.
`compare`, `tags` and `pin` belong on groups and evals, not in the config.
## Errors
Every config error exits 2 before anything runs. An unknown key gets a hint when it looks like a known one:
```text
reeval: Unknown option "harnesses" in reeval.config.ts. Did you mean "harness"?
```
| Mistake | Message |
|---|---|
| `compare` in the config | `compare can't be set in reeval.config.ts: it would multiply every eval. Put it on the group or eval that asks the question.` |
| A literal secret | `env.API_KEY in reeval.config.ts is a literal string. Pass secrets with secret.fromEnv() so they're redacted and never recorded.` |
| A judge model object | `judge.model in reeval.config.ts must be a "provider/model" string or fakeJudge(). Pass a model object per group, eval or call instead.` |
| A model without a provider | `judge.model in reeval.config.ts needs a provider prefix, e.g. "typesafe-ai/jev-1.14.0" or "google/gemini-3.8-flash" (got "gemini-3.8-flash")` |
| A bad duration | `Invalid duration "10 minutes". Use e.g. "90s" or "15m".` |
| A zero or negative duration | `Invalid duration 0. A duration is more than zero: use e.g. "90s" or "15m".` |
| An empty `gateway` list | `gateway in reeval.config.ts must be a list of gateways in the order you prefer them, e.g. gateway: [direct()] (got [])` |
| A bad concurrency | `concurrency in reeval.config.ts must be a whole number of at least 1 (got 0)` |
| An unknown `debug` key | `Unknown option "keepRuns" in debug in reeval.config.ts. Use keep, traceBufferMb.` |
| `analysis.insights` that isn't `true` or `false` | `analysis.insights in reeval.config.ts must be true or false (got "yes")` |
| A token budget below 1 | `analysis.maxTokens in reeval.config.ts must be a whole number of tokens of at least 1, e.g. 250000 (got 0)` |
| A missing default profile | `defaultProfile.ci in reeval.config.ts names "pr", which isn't a profile (it defines none). Add it under profiles, or name one that's there.` |
| A `Promise` from the callback | `The defineConfig callback in reeval.config.ts returned a Promise. …` |
| No `reeval.config.*` here or above | `No reeval.config.ts (or reeval.config.mts, reeval.config.cts, reeval.config.js, reeval.config.mjs, reeval.config.cjs, reeval.yaml, reeval.yml) here or above. Run reeval init.` |
| A `reeval.yaml` beside another config | `reeval.config.ts and reeval.yaml are both in .: reeval reads one config per directory. Keep one, or pass --config.` |
| A YAML syntax error | `reeval.yaml isn't valid YAML: Map keys must be unique at line 2, column 1: …` |
| A Dockerfile written as data that isn't there | `sandbox.dockerfile in reeval.yaml: /project/docker/Dockerfile doesn't exist. Paths in reeval.yaml are relative to its directory.` (or `reeval.config.ts`) |
| A `null` or a misspelt nested key in `reeval.yaml` | `include in reeval.yaml must be a list (got null)`, `Unknown option "dirr" in history in reeval.yaml. Did you mean "dir"?` |
More messages, with fixes, are on [Errors](/reference/errors).
## Related
- [Options](/reference/options): how each option key cascades.
- [Profiles and budgets](/reference/profiles)
- [Helpers](/reference/helpers)
- [Environment variables](/reference/env-vars)