---
title: group and evaluate
description: >-
  The signatures of group and evaluate, their skip, only, todo and each
  variants, name placeholders, families and eval ids.
---
> ## Documentation Index
> Fetch the complete documentation index at: https://docs.reeval.ai/llms.txt
> Use this file to discover all available pages before exploring further.

`evaluate` declares one eval; `group` names a set of evals and gives them shared options.

```ts title="evals/math.eval.ts"
import { evaluate, expect, group } from '{{ pkg }}'

group('math', { tags: ['smoke'] }, () => {
  evaluate('writes add.js', { timeout: '5m' }, async (c) => {
    const turn = await c.send('Create add.js exporting add(a, b).')
    await expect(turn).toHaveFile('add.js')
  })

  evaluate.each([
    { a: 2, b: 3 },
    { a: 10, b: -4 },
  ])('adds $a and $b', async (c, { a, b }) => {
    await c.send(`Create sum.txt containing ${a} + ${b}.`)
  })

  evaluate.skip('subtracts', async () => {})
  evaluate.todo('multiplies')
})
```

```text
plan  3 evals · 3 cells · 3 attempts · …
eval                                     harnesses  variants  runs  attempts  notes
ge/math.eval.ts › math › writes add.js   1          —         1     1
ge/math.eval.ts › math › adds $a and $b  1          —         1     2         2 cases · pooled pass rate
ge/math.eval.ts › math › multiplies      —          —         0     0         todo
```

Eval files match `include` in `reeval.config.ts` (default `evals/**/*.eval.ts`). Both functions come from `'{{ pkg }}'`.

## evaluate

```ts
evaluate(name, fn)
evaluate(name, options, fn)
evaluate(name, fn, options)
```

<Property name="name" type="string">
  The eval's name. Together with its file and groups it makes the eval id. Name it after the question it answers, such as `recommends Genkit for a production AI feature` rather than `genkit test 2`: the run's insights read the name first to tell what the eval checks.
</Property>

<Property name="options" type="EvalOptions" optional>
  A plain object of options: `harness`, `environment`, `compare`, `sandbox`, `runs`, `gate`, `env`, `egress`, `judge`, `timeout`, `tags`, `pin`. See [Options](/reference/options).
</Property>

<Property name="fn" type="(c: AttemptContext) => Promise<void>">
  The eval body. It runs once per attempt with a fresh `c`. It passes when it returns without throwing. See [The attempt context](/reference/attempt-context).
</Property>

Every cell of the eval (harness × environment × sandbox, plus `compare` variants) runs `runs` attempts of `fn`.

## group

```ts
group(name, fn)
group(name, options, fn)
group(name, fn, options)
```

<Property name="name" type="string">
  The group's name, shown before its evals' names.
</Property>

<Property name="options" type="GroupOptions" optional>
  The same keys as `evaluate`. Evals inside inherit them; an inner group or eval overrides them key by key, following the merge rules on [Options](/reference/options).
</Property>

<Property name="fn" type="() => void">
  Declares the group's evals and nested groups. It runs while reeval collects evals, so keep it synchronous and free of side effects.
</Property>

## Variants

| Call | What it does |
|---|---|
| `evaluate.skip(name, …)` | Declares the eval but doesn't run it; it counts as skipped. |
| `evaluate.only(name, …)` | Runs only the `.only` evals and groups of its file. |
| `evaluate.todo(name)` | A placeholder with no body. It's one entry marked todo: a `todo` row in `--dry-run` (and `mode: 'todo'` under `excluded` in `--json`), `todo` in the summary and `summary.md`, `todo: true` with outcome `skipped` in the record, and Todo in the studio. It counts as skipped. |
| `evaluate.each(cases, options?)(name, …)` | One eval per case, pooled into a family. The body gets `(c, item)`. |
| `group.skip(name, …)` | Skips every eval in the group, nested groups included; each counts as skipped. |
| `group.only(name, …)` | Runs only this group's evals within its file. |
| `group.todo(name)` | A placeholder group, shown exactly as `evaluate.todo(name)` is: one todo entry, with the id `file › …groups › name`. |

`.only` is refused when `CI` is set, so a forgotten `.only` can't shrink a CI run: `evaluate.only isn't allowed in CI` (or `group.only`).

## evaluate.each

```ts
evaluate.each(cases)(name, fn)
evaluate.each(cases)(name, options, fn)
evaluate.each(cases, { pool: false })(name, options, fn)
```

<Property name="cases" type="readonly T[]">
  The cases. Each one becomes an eval, and `fn` gets it as its second argument: `(c, item)`.
</Property>

<Property name="pool" type="boolean" optional>
  Pool the cases into a family. Default `true`; a family needs two or more cases. `false` keeps them independent evals.
</Property>

`name`, `options` and `fn` are those of `evaluate`, and the options apply to every case.

### Names

`evaluate.each` fills placeholders in the name from each case:

| Placeholder | Becomes |
|---|---|
| `$key` | That property of the case (`$a` → `2`) |
| `%s` | The case itself: a string as is, anything else as JSON |
| `%#` | The case's index, from 0 |

`evaluate.each([{ a: 1, b: 2 }, { a: 3, b: 4 }])('adds $a and $b (case %#)', …)` declares `adds 1 and 2 (case 0)` and `adds 3 and 4 (case 1)`. A `$key` the case doesn't have stays as written.

Two cases that end up with the same name stop the run with exit 2 (`evaluate("adds 1") is declared twice in this file. …`), as do two `evaluate` calls with the same name in the same group and file. Each eval's attempts belong to its id, so they'd otherwise merge.

### Families

A pooled `evaluate.each` is a family, with the id `file › …groups › name template` (`ge/math.eval.ts › math › adds $a and $b`). Each case keeps its own eval id, cells, outcome and history; the family adds:

| What | Where |
|---|---|
| One plan row, with `N cases · pooled pass rate` (or `paired lift` with `compare`) | `--dry-run`; `--json` lists every case with its `family` |
| One result block: a row per case under the template and, with `compare`, an `all cases` row with each arm's pooled count and each variant's paired effect | `reeval run`, `summary.md`, `reeval show` |
| A pooled pass@1 per cell, with its interval | `reeval query families`, the studio's run page |
| A gate: a variant's pooled lift whose lower bound is below `minLift` fails the run | Exit 1, `not by minLift N` on the `all cases` row |
| `evals[].family` and `families[]` | The [run record](/reference/record) |
| `family_lift` | [`reeval query`](/reference/cli/query) |

`minLift` on a family gates its pooled lift only, not each case's. Renaming the template starts a new family. See [Suites of cases](/recipes/eval-each) and [Lift](/concepts/lift).

## Eval ids

An eval's id is its file path (relative to the config's directory), its groups and its name, joined with ` › `:

```text
ge/math.eval.ts › math › writes add.js
```

The id is what the plan, the reporters, `reeval history` and the run record show, and what `reeval run` filters match. Each attempt also has an id, `<cell key>#<n>`, such as `cursor/-/docker#1`.

## Eval files

```ts title="evals/greet.eval.ts"
import { readFile } from 'node:fs/promises'
import { evaluate } from '{{ pkg }}'
import cases from './cases.json'            // JSON imports
import { house } from './style'             // extensionless: style.ts
// Top-level await works too.
const brief = await readFile(new URL('./brief.md', import.meta.url), 'utf8')
```

Eval files are TypeScript or JavaScript, ESM or CommonJS, loaded with no build step. Imports may leave out the extension, and JSON imports work. Each eval file loads in a fresh process of its own, so module state and globals never leak from one file to another; `reeval run` loads each file twice, once for the plan and once to run.

The nearest `tsconfig.json` must load: one that isn't valid JSON (comments and trailing commas are fine) or extends a missing file stops the run. Its `paths` aliases aren't applied.

Decorators are TypeScript's legacy ones. npm, pnpm, Yarn 1 and Yarn's Plug'n'Play all work.

## Timeouts

```ts
evaluate('build', {
  timeout: '5m',
  harness: cursor({ startTimeout: '2m' }),
}, async (c) => {
  // …
})
```

`timeout` covers one attempt's body. Starting the sandbox, running setup steps and starting the harness pause the clock; the harness's start has its own `startTimeout` (default 90s, retried once). An attempt that runs out fails with `Eval timed out after 300s`, and `c.signal` aborts so you can stop work early.

## Errors

| Mistake | Message |
|---|---|
| Two functions | `evaluate("x") got two functions. Options must be a plain object; helpers such as secret are only available in defineConfig and defineEnvironment callbacks. …` |
| An unknown option | `Unknown option "repeat" in evaluate("e"). Did you mean "runs"?` |
| `.only` in CI | `evaluate.only isn't allowed in CI` |
| Eval options as `evaluate.each`'s second argument | `evaluate.each(cases, options) takes { pool?: boolean } (got {"compare":{}}). Eval options such as compare go after the name: …` |
| `pool` on an eval | `Unknown option "pool" in evaluate("x"). pool is an option of evaluate.each, beside the cases: evaluate.each(cases, { pool: false }).` |

Errors in an eval file stop the run with exit 2 and name the file. More on [Errors](/reference/errors).

## Related

- [Writing evals](/guide/writing-evals)
- [Options](/reference/options)
- [The attempt context](/reference/attempt-context)
- [Matchers](/reference/matchers)