---
title: Judges
description: >-
Judge model strings, how reeval picks a provider and key, thresholds,
concurrency, retries and fakeJudge.
---
> ## Documentation Index
> Fetch the complete documentation index at: https://docs.reeval.ai/llms.txt
> Use this file to discover all available pages before exploring further.
`c.judge(question)` asks a model a yes/no question about the latest turn and returns a verdict. `expect(verdict).toPass()` fails the eval when the answer is no.
```ts title="evals/greet.eval.ts"
import { evaluate, expect, group } from '{{ pkg }}'
group('greet', { judge: { threshold: 0.9 } }, () => {
evaluate('explains its change', async (c) => {
await c.send('Create greet.js exporting greet(name).')
const verdict = await c.judge(
"Does the agent's final message describe what it changed?",
)
expect(verdict).toPass()
})
})
```
## Settings
`judge` is accepted in `reeval.config.ts`, profile `judge` and `defaults.judge`, groups, evals, and as the second argument of `c.judge()`. It merges field by field, the most specific level winning: the call, then the eval, then its groups, then the profile, then the config.
`"provider/model"`, such as `typesafe-ai/jev-1.14.0` (reeval's recommended judge) or `google/gemini-3.8-flash`. `fakeJudge()` for tests. Groups, evals and calls also accept an AI SDK decision model object. There's no default: `c.judge()` without a model fails.
The score, from 0 to 1, at or above which the verdict is `pass`. Default `0.8`.
Passed to the provider as is, keyed by provider name.
The most judge calls at once to this model's provider, across the run. A whole number of at least 1.
The gateway the judge's calls go through: a gateway's name (`'vercel'`) or its helper (`vercel()`). It skips the config's `gateway` fallback, and must name a gateway in the list that can serve the model. Without a pin or a list, the judge goes `direct()`, on its provider's own key. See [Gateways](/reference/gateway#the-judge-analysis-and-captures).
What the judge is given beside your question: the judge's state, not the workspace's or the attempt's. An object keyed by the state's fields, described in [What the judge is given](#what-the-judge-is-given). It merges key by key, so a call's `state: { transcript: 'tools' }` keeps the config's `diff` patterns. A key's value replaces the outer one whole; lists aren't concatenated.
`c.judge()` also takes `on`: a `Turn` this attempt's `c.send()` returned, to grade instead of the latest, or any JSON value to grade as the output (with the latest turn's task, files and diff). Only those `Turn` objects count as turns: a plain object shaped like one is a value, and a copy of a turn (`{ ...turn }`) is refused. `on` with a value and `state: { output: false }` is refused.
`artifacts` is call-only: `true` for every capture so far, or a list of `c.capture()` handles or their names. Each capture is sent as a file part. Jev (`typesafe-ai/jev-1.14.0`, with `TYPESAFE_AI_API_KEY`) takes PNG captures; a provider that cannot take files fails the call with its own error.
Captures this call sends the judge: `true` for every capture the attempt has taken so far, in the order it took them, or a list of handles from `c.capture()` or capture `name`s. The PNG is a file part (`{ type: 'file', mediaType: 'image/png', data }`). Ask about facts in the picture, not whether it "looks great". JSON-only calls (no `artifacts`) stay a JSON object.
## What the judge is given
```ts
const verdict = await c.judge('Does sign-in handle a wrong password?', {
state: { diff: ['src/', '!**/*.test.ts'], contents: ['README.md'] },
})
```
The judge gets your question and a JSON state, which `state` chooses. It's in `judge-input.json` as each call's `state`; the resolved option is beside it as `stateOptions`.
With `artifacts`, the state is an ordered list: one JSON part, then each capture as a file part. The score is the model's probability that the answer is yes.
| `state` key | Type | Default | What the state holds |
|---|---|---|---|
| `task` | `boolean` | `true` | The graded turn's prompt |
| `output` | `boolean` | `true` | The graded turn's final message, or `on`'s value |
| `files` | `boolean` | `true` | The changed files, `status path`, filtered by `diff`'s patterns; dependency and build directories as one line each |
| `diff` | `boolean \| string[]` | `true` | The diff since the eval started, without the paths in `contents`. A list is gitignore-style patterns over workspace paths, matched like [`directory()`'s `glob`](/reference/helpers#directory): the last match wins and `!pattern` excludes |
| `contents` | `string[]` | `[]` | Workspace files whole, as they are when you call `c.judge()`, changed or not, keyed by path in path order. Exact paths or the same patterns as `diff`. A missing exact path is `null`; a pattern that matches nothing adds nothing |
| `transcript` | `boolean \| 'tools' \| 'full'` | `false` | The graded turn's transcript as text lines: `true` messages and tool calls, `'tools'` tool calls only, `'full'` reasoning and unclipped tool output too |
| `turns` | `'last' \| 'all'` | `'last'` | `'all'` moves `task`, `output` and `transcript` into `turns`, one entry per turn up to the graded one |
| `checks` | `boolean \| string[]` | `false` | The `c.exec` results so far as `{ command, exitCode, stdout, stderr }`; a list keeps only those commands, as passed to `c.exec` |
| file parts | | | With `artifacts`: each capture as `{ type: 'file', mediaType: 'image/png', data }` |
A key that's off leaves its field out of the state. Patterns in `contents` expand over the workspace as git sees it, so its `.gitignore` applies, and a binary file's value is `[reeval: dropped binary file]`. Transcript lines clip each tool's output to 300 characters, except under `'full'`:
```text
tool shell: npm test → ok · 0ms
14 passing
edit src/app.ts (create, +1 −0) · 1ms
```
The diff leaves out what nobody wrote by hand, each file or directory replaced by a `[reeval: dropped …]` marker. To show one anyway, name it in `contents`.
| Left out | Matched by |
|---|---|
| Installed dependencies | `node_modules/`, `.venv/`, `.dart_tool/`, `.pub-cache/`, `__pycache__/` |
| Build output | `dist/` and `.next/` at any depth; `build/` and `out/` at the top of the workspace |
| Lockfiles | `package-lock.json`, `pnpm-lock.yaml`, `yarn.lock`, `bun.lock`, `bun.lockb` |
| Generated files | `*.map`, `*.tsbuildinfo` |
| Minified files | A changed line of more than 1,000 characters |
| Binaries | Anything git diffs as binary |
What was left out, and why, is the judge input's `dropped` in `judge-input.json`, such as `dist/ (3 files): build output` or `notes.md: not in state.diff`. The workspace's own `.gitignore` keeps the rest of the noise out of the diff; see [Environment](/reference/environment#fields).
## Provider errors
```text
judge failed for "Is it done?" (typesafe.evaluation/jev-under-test): {"error_type":"max_tokens_exceeded"}
The judge's input was ~22k tokens (estimated): diff 19k, of which src/pages/SignIn.tsx is 17k, src/firebase.ts is 2k; transcript 3k. Narrow state.diff, state.transcript, state.files, state.task or state.output.
```
reeval sends the judge everything `state` selects, uncut. When the provider refuses it, or a judge call fails with any other error from the provider's API, the message keeps the provider's wording and adds one line.
That line has the input's estimated size, at four characters a token, its biggest parts and files, and the `state` keys that were on, largest first. reeval knows no model's limit and doesn't read the provider's message, so the line comes with every provider error.
## Failed calls
A call that fails (a provider error after its retries, or a missing key) doesn't throw. `c.judge()` returns a `JudgeFailure`, the call's error is recorded with its input, and the judges and checks after it still run. When the eval's body is done, the attempt fails as a `judge` error with every failed call's message.
```ts
{ verdict: 'error', score: null, threshold, question, model, error }
```
`toPass()` and `.not.toPass()` both fail on it, with the call's error. `c.judge()` still throws on a mistake in the call itself (no model, an empty question, a bad threshold, an unknown option or `state` key, a `state.checks` command it hasn't run) and when the run is cancelled or times out.
## Providers and keys
```ts title="reeval.config.ts"
import { defineConfig } from '{{ pkg }}'
import { vercel } from '{{ pkg }}/gateway'
export default defineConfig({
judge: { model: 'typesafe-ai/jev-1.14.0' }, // TYPESAFE_AI_API_KEY
// gateway: [vercel()], // to take AI_GATEWAY_API_KEY instead
})
```
reeval uses no gateway by default. Without a [`gateway` list](/reference/gateway#the-judge-analysis-and-captures), a judge calls its provider's own package with its provider's key, and `AI_GATEWAY_API_KEY` is never read. With a list, the judge takes the first gateway that serves its model, or the one it pins, and only that gateway's key.
| Provider prefix | `direct()` key (the default) | Package |
|---|---|---|
| `anthropic/` | `ANTHROPIC_API_KEY` | `@ai-sdk/anthropic` |
| `google/` | `GOOGLE_GENERATIVE_AI_API_KEY` | `@ai-sdk/google` |
| `openai/` | `OPENAI_API_KEY` | `@ai-sdk/openai` |
| `typesafe-ai/` | `TYPESAFE_AI_API_KEY` | `@ai-sdk/typesafe-ai` |
| Any other | None: needs `vercel()` or `openrouter()` in `gateway` | The AI Gateway, with `AI_GATEWAY_API_KEY` or `VERCEL_OIDC_TOKEN`; or `@openrouter/ai-sdk-provider`, with `OPENROUTER_API_KEY` |
A judge answers through a decision API, so two routes take only decision models. An `openai/` judge calls OpenAI's Decisions API, which answers with `openai/gpt-6-luna`; a language model such as `openai/gpt-5.5` fails each call. On `vercel()`, a judge must be one of Vercel's decision models, such as `typesafe-ai/jev-1.14.0` or `openai/gpt-6-luna-decisions`; on `openrouter()`, one OpenRouter answers on its Decisions API, such as `typesafe/jev-1.14` (see [Gateways](/reference/gateway#the-judge-analysis-and-captures)). On OpenRouter each judgement also records the `upstream` provider that served it and its `costUsd`. `anthropic/` and `google/` judges take their language models on `direct()`.
`analysis.model`, insights and questions route the same way. They need a language model, though, so an evaluation model such as `typesafe-ai/jev-1.14.0` can judge but can't write an analysis.
The run checks the judge's key before it starts, with the harness's. When the run's profile sets a judge `model`, that judge's key is the one checked, and a `fakeJudge()` needs none. The message names the model and the key its gateway takes; when `AI_GATEWAY_API_KEY` is set but no gateway uses it, it says how to:
```text
reeval: Missing credentials: CURSOR_API_KEY; ANTHROPIC_API_KEY (for direct()) for the judge model, "anthropic/claude-sonnet-5". Set them in the environment or .env.local. AI_GATEWAY_API_KEY is set, but no gateway in the config uses it: add vercel() to gateway to call the model through Vercel AI Gateway.
```
## Retries
| Retried | How |
|---|---|
| 408, 409, 425, 429 and any 5xx | Up to six tries in all |
| A message about high demand, overload, rate limits or a reset connection | The same |
| The wait between tries | Exponential from 2 seconds, capped at 30, and at least any `retry-after` |
Each retry is recorded on the verdict as `{ error, delayMs }`. A judge that still fails is a [failed call](#failed-calls): a judge error for the attempt, not a failed check.
## fakeJudge
```ts
import { fakeJudge } from '{{ pkg }}/judge'
judge: { model: fakeJudge({ score: 0.95 }) }
```
A judge that returns `score` (default `1`) for every question, with no key and no call. Its model name is `fake`.
Per group, eval or call, `score` can be a function of the state the judge would see, to test what reaches it: `fakeJudge({ score: (state) => (String(state.diff).includes('dist/') ? 0 : 1) })`. The config takes a number.
`'{{ pkg }}/judge'` also exports `RECOMMENDED_JUDGE_MODEL` (`'typesafe-ai/jev-1.14.0'`) and `DEFAULT_THRESHOLD` (`0.8`).
## fakeAnalysis
```ts
import { fakeAnalysis } from '{{ pkg }}/judge'
analysis: { model: fakeAnalysis({ output: { … } }) }
```
The same idea for `reeval analyze`: it answers with `output`, or `output(digest)` when you pass a function, and reeval parses and validates it as it would a real model's answer. Use it to test the analysis without a key. See [reeval analyze](/reference/cli/analyze).
```ts
analysis: {
model: fakeAnalysis({
vote: (pair) => (pair.a.diff?.includes('Ahoy') ? 'A' : 'tie'),
}),
}
```
`vote` answers pairwise [questions](/guide/questions): `'A'` (the attempt shown first), `'B'` or `'tie'` for every pair, `{ winner, reason? }` to record a reason, or a function of the blinded pair (`{ question, prompt, a, b }`). `a` and `b` are the two sides in the order shown, each holding only the parts its question's [`state`](/reference/options#question) turned on: `task?`, `output?`, `transcript?`, `turns?`, `files?`, `diff?`, `checks?` and `artifacts?` (`{ label, mediaType, bytes }`, or `{ label, missing: true }`). `prompt` is the text the model reads, with a task both sides share. `{ error: '…' }` fails every call, to test an answer's `error`.
```ts
analysis: {
model: fakeAnalysis({
answer: (q) => ({ choice: q.attempt.diff?.includes('Ahoy') ? 'yes' : 'no' }),
}),
}
```
`answer` answers [questions](/guide/questions) asked of each attempt: a string for a free-text answer, `{ choice?, text? }` (a `choice` must be one of the question's), or a function of the blinded attempt (`{ question, prompt, choices?, attempt }`), where `attempt` holds the parts its `state` turned on, like a pair's side. Without it, a question with `choices` gets the first and a free-text one a fixed sentence. `{ error: '…' }` fails every call.
```ts
analysis: {
model: fakeAnalysis({
insights: (input) =>
input.call === 'classify'
? { intent: { job: 'finish', question: 'Does it build?' } }
: input.call === 'insights'
? { answer: { text: 'Yes: every try passed.', cites: [input.facts[0].id] } }
: { headline: null },
}),
}
```
`insights` answers the run's [insights](/reference/cli/analyze#insights), at the end of `reeval run` and for `reeval analyze --insights`: fixed insights, or a function (or an async one) of each call's input. `input.call` is `classify` (the eval's name, prompts, judge questions, check labels and facts; answer `intent`), `insights` (the same, with `intent`, `slots`, the rows that take a sentence, and `source`, the eval's declaration as `L` lines; answer `answer`, `findings`, `rows`, `namedTools` and `remarks`), or `headline`, once a run (answer `headline`). Each fact, attempt and eval carries the short `id` the model cites (`F1`, `A1`, `A1#L12` for a transcript line), and `slots.rows` lists fact ids. Every sentence goes through the validator, and every remark through the [remark gates](/reference/cli/run#model-remarks) (`source.at` and `fix.at` are `L` lines, `evidence` fact or attempt ids), so a fake sentence that cites nothing is dropped as a model's would be; `{ error: '…' }` fails every call.
Without `vote`, every pair is a tie; without `output`, `reeval analyze` refuses the fake; without `insights`, insight calls fail, and `reeval run` prints `insights failed` with its facts.
## Errors
| Mistake | Message |
|---|---|
| No model | `c.judge() has no model. Set judge: { model: 'typesafe-ai/jev-1.14.0' } in reeval.config.ts, on the group or eval, or pass { model } to this call.` |
| No turn yet | `c.judge() has no turn to grade: call c.send() first` |
| No provider prefix | `judge.model in reeval.config.ts needs a provider prefix, e.g. "typesafe-ai/jev-1.14.0" or "google/gemini-3.8-flash" (got "gemini-3.8-flash")` |
| A model object in the config | `judge.model in reeval.config.ts must be a "provider/model" string or fakeJudge(). Pass a model object per group, eval or call instead.` |
| A bad threshold | `judge.threshold in reeval.config.ts must be between 0 and 1 (got 2)`, or `in group("g")` on a group or eval; on a call, `c.judge() threshold must be between 0 and 1 (got 2)` |
| An empty question | `c.judge() needs a question: a non-empty string (got "")`, before the judge is called |
| A bad `maxConcurrent` | `judge.maxConcurrent in evaluate("x") must be a whole number of at least 1 (got 0)` |
| An unknown option | `Unknown judge option "diff" in reeval.config.ts. Did you mean "state.diff"?`; on a call, `… in c.judge(). …`. `on` outside a call: `on is per call: c.judge(question, { on }).` |
| An unknown `state` key | `Unknown judge.state key "transcipt" in reeval.config.ts. Did you mean "transcript"?` |
| A bad `state` value | `judge.state.turns in evals/a.eval.ts › lands must be 'last' or 'all' (got "first")`; `state` itself must be an object such as `{ transcript: 'tools' }` |
| A check it hasn't run | `c.judge() state.checks names a command this attempt hasn't run: "npm tset"` |
| `on` with no output | `c.judge() on gives the output, but state.output is false` |
| A bad `artifacts` | `c.judge() artifacts must be true or a list of capture handles or names (got "form")` |
| An unknown capture name | `c.judge() artifacts names a capture this attempt doesn't have: "form"` |
| A score function in the config | `judge.model in reeval.config.ts must be fakeJudge({ score }) with a number. Pass a score function per group, eval or call instead.` |
| The provider refuses the input | The provider's message, then `The judge's input was ~29k tokens (estimated): diff 28k, of which …`, as a failed call |
| An OpenAI language model as a judge | `the judge model "openai/gpt-5.5" isn't a decision model: reeval's openai/* judges call OpenAI's Decisions API, which answers with 'gpt-6-luna' (OpenAI said: …). Use 'openai/gpt-6-luna'.`, as a failed call |
| A language model as a judge on `vercel()` | `the judge model "anthropic/claude-sonnet-5" is a language model, and Vercel AI Gateway judges only with decision models. …`, as a failed call |
| No key at call time | `No credentials for judge model "google/gemini-3.8-flash" (provider "google"). Set GOOGLE_GENERATIVE_AI_API_KEY (for direct()), in the environment or .env.local.` |
| A Gateway key set, but no `vercel()` in `gateway` | `… Set TYPESAFE_AI_API_KEY (for direct()), in the environment or .env.local. AI_GATEWAY_API_KEY is set, but no gateway in the config uses it: add vercel() to gateway to call the model through Vercel AI Gateway.` |
| A judge OpenRouter doesn't answer as a decision model, on `openrouter()` | `the judge model "anthropic/claude-haiku-4.5" isn't a decision model OpenRouter answers on its Decisions API (OpenRouter said: …). Use a decision model on openrouter(), such as 'typesafe/jev-1.14'.`, as a failed call |
| A prefix without a direct route, and no `vercel()` | `evals/a.eval.ts › e: the judge model "mistral/large": no gateway in the list can serve it (direct: reeval calls only anthropic/*, google/*, openai/*, typesafe-ai/* models on their own keys). Add vercel() to gateway. There a judge must be a decision model, such as 'typesafe-ai/jev-1.14.0'.`, at plan time |
| A key with whitespace or a line break | `ANTHROPIC_API_KEY for judge model "anthropic/claude-sonnet-5" has whitespace or a line break in it at character 7. …` |
## Related
- [Judges concepts](/concepts/judges)
- [The attempt context](/reference/attempt-context)
- [Matchers](/reference/matchers)
- [Writing evals](/guide/writing-evals)