--- title: Judges description: >- How a judge scores a yes/no question, which model it calls, and how to write questions it can grade. --- > ## Documentation Index > Fetch the complete documentation index at: https://docs.reeval.ai/llms.txt > Use this file to discover all available pages before exploring further. `c.judge(question)` asks an evaluation model a yes/no question about the agent's latest turn. The model returns the probability of "yes", and the verdict passes when that score is at or above the threshold. ```ts title="evals/hello.eval.ts" evaluate('adds a default name in a follow-up turn', async (c) => { await c.send( 'Create greet.js exporting greet(name) that returns "Hello, !". ' + 'No dependencies.', ) await c.send( 'Make name default to "world" when greet() is called without arguments.', ) await expect(c.exec(callGreetWithNoArgs)).toSucceed() expect( await c.judge( 'Did the agent add the default without breaking greet(name)?', ), ).toPass() }) ``` ```text [adds#1 cursor] +0:10 judge pass 1.00 (≥ 0.8) Did the agent add the default without breaking greet(name)? ``` ## Score and threshold The score is P(yes), from 0 to 1. The threshold defaults to 0.8, and a verdict is `{ verdict, score, threshold, question, model }`. With `-v`, the judge line says why: ```text [adds#3 fake] 17:15:24 judge fake · 4ms · score 0.90 ≥ 0.7 · pass · reason: P(yes) 0.90 is at or above the 0.7 threshold ``` `judge` merges field by field, innermost first: the call, then the eval, then each group, then the config. So an eval or a single call can change the threshold and keep the config's model: ```ts title="evals/strict.eval.ts" evaluate('strict', { judge: { threshold: 0.9 } }, async (c) => { await c.send('Rename add.js to sum.js and update its imports.') const minimal = await c.judge('Is the change minimal?', { threshold: 0.7 }) expect(minimal).toPass() }) ``` ## Which model it calls There's no default model. Set `judge.model` in the config, a group, an eval or the call, or `c.judge()` fails with `c.judge() has no model. Set judge: { model: 'typesafe-ai/jev-1.14.0' } in reeval.config.ts, on the group or eval, or pass { model } to this call.` ```mermaid flowchart LR M["model string"] --> L{"vercel() in gateway, first to serve it?"} L -- no --> D{"provider's own key set?"} D -- yes --> P["provider's AI SDK package"] D -- no --> E["the run stops before it starts, naming the key"] L -- yes --> W["Vercel AI Gateway, with AI_GATEWAY_API_KEY"] ``` reeval uses no gateway by default: a judge takes its provider's own key. The Vercel AI Gateway is used only when the config's [`gateway` list](/reference/gateway#the-judge-analysis-and-captures) has `vercel()`: first in the list, or pinned with the judge's `gateway: 'vercel'`. | `model` | Key, with no `gateway` list | With `vercel()` first in `gateway` | |---|---|---| | `'typesafe-ai/jev-1.14.0'` (recommended) | `TYPESAFE_AI_API_KEY` | `AI_GATEWAY_API_KEY` | | `'google/gemini-3.8-flash'` | `GOOGLE_GENERATIVE_AI_API_KEY` | `AI_GATEWAY_API_KEY` | | `'anthropic/…'` | `ANTHROPIC_API_KEY` | `AI_GATEWAY_API_KEY` | | `'openai/…'` | `OPENAI_API_KEY` | `AI_GATEWAY_API_KEY` | | Any other `provider/model` | Refused at plan time: add `vercel()` to `gateway` | `AI_GATEWAY_API_KEY` | | An evaluation model object | Used as is, per group, eval or call only | — | | `fakeJudge({ score })` | A fixed score, no model call | — | Pin model versions: `jev-1.14.0`, not `jev-latest`. A moving alias changes scores between runs, and the judge's model is part of every [fingerprint](/concepts/fingerprints). ## Retries A judge call that hits a transient error (a 429, a 5xx, "high demand", a dropped connection) is retried with exponential backoff, up to six tries over about a minute, honouring `retry-after`. A bad request or a bad key fails straight away. | Where | What's recorded | |---|---| | The verdict | `retries`: each error and the delay before the next try | | The attempt | A judge call that fails for good returns `verdict: 'error'`; the judges and checks after it still run, then the attempt is a `judge` failure | | The timeout | Retries count toward the eval's `timeout` | `judge: { maxConcurrent: 2 }` caps calls at once per provider, for rate limits. See [Concurrency](/concepts/concurrency). ## What the judge is given By default the judge is given four things, its state: the latest turn's prompt as `task`, the agent's final message as `output`, the changed files and the diff since the eval started. Each call is saved in the attempt's `judge-input.json`. From a real Cursor attempt of the multi-turn eval above: ```text [ { "question": "Did the agent add the default without breaking greet(name)?", "model": "google/gemini-3.8-flash", "state": { "task": "Make name default to \"world\" when greet() is called without arguments.", "output": "Updated `greet` to use a default parameter: `name = \"world\"`. Calling `greet()` returns `\"Hello, world!\"`; passing a name still works as before.", "files": [ "added greet.js" ], "diff": { "$ref": "diff.patch", "sha256": "2d8d95c242d9ea1246dd52368f13f5220d8992713885b2be946af5c939915fdc" } } } ] ``` | Given by default | Given only when `state` asks | |---|---| | The latest turn's prompt | Earlier turns: `turns: 'all'` | | The agent's final message | The transcript: `transcript: true`, `'tools'` or `'full'` | | Changed files, dependency and build folders collapsed | Unchanged files whole: `contents` | | The diff since the eval started, without build output, lockfiles, source maps and minified files | Your checks and their output: `checks` | So by default the judge can't tell whether the agent ran the tests; check that with `c.exec`, or show it the transcript. `{ on: turn }` grades an earlier turn, and `{ on: value }` grades any value in place of the final message. `{ artifacts: [image] }` adds a `c.capture()` PNG as a file part; ask about facts in the picture, not whether it "looks great". ## Judge what the code prints ```ts title="evals/hygiene.eval.ts" const run = await c.exec('node summary.js') const verdict = await c.judge('Does it say 3 orders totalling 50?', { on: run.stdout, }) ``` ```text "question": "Does it say 3 orders totalling 50?", … "task": "Write summary.js. It prints the order count and total of orders.csv.", "output": "3 orders, total 50.00\n", "files": [ "added greet.js", "added summary.js" ], … ``` The diff shows the code, not what it does. When the behaviour is the outcome, run the code and pass its output with `on`: in `judge-input.json` it replaces the final message as `output`, and the task, files and diff stay. [Make the eval trustworthy](/guide/writing-evals#make-the-eval-trustworthy) has the whole eval. ## Choose what the judge is given ```ts title="evals/sign-in.eval.ts" const flow = await c.judge('Does this change implement email sign-in?', { state: { diff: ['src/', '!**/*.test.ts'], contents: ['src/firebase.ts'] }, }) expect.soft(flow, 'Email sign-in').toPass() ``` `state` is what the judge is given, not the workspace's or the attempt's state: it picks the evidence beside the graded output, and its keys are the fields of the state in `judge-input.json`. `diff` patterns keep the diff to the files a question is about, and `contents` shows files whole, changed or not. A smaller state fits the model's context, and the judge grades better without the noise. ```ts c.judge('Is this code idiomatic?', { state: { task: false } }) c.judge('Is the change correct?', { state: { output: false } }) c.judge('Can a user add a todo?', { state: { contents: ['src/**'] } }) ``` Leave the prompt out when it would anchor a generic question, and the final message when a confident summary shouldn't sway a judge grading the diff. `contents` takes patterns too, for when you can't know the filenames. `state` cascades like every other option, key by key. The diff is what changed in the workspace, so give the workspace its own `.gitignore`: an agent that runs `npm run build` leaves its output there. reeval drops known build folders, and sends the rest as it is: a state too big for the model fails the call with the provider's error. ## Show the judge what the agent did ```ts title="evals/tests.eval.ts" await c.exec('npm test') const ran = await c.judge('Did the agent run the tests before finishing?', { state: { transcript: 'tools', checks: ['npm test'] }, }) ``` ```text "checks": [ { "command": "npm test", "exitCode": 0, "stdout": "\n> test\n> echo 14 passing\n\n14 passing\n", "stderr": "" } ], "transcript": "tool shell: npm test → ok · 0ms\n 14 passing\nedit src/app.ts (create, +1 −0) · 2ms\n…" ``` `transcript` adds the graded turn's tool calls as text lines, and `checks` adds your `c.exec` results with their exit codes and output. `turns: 'all'` gives every turn's prompt and final message, for a question about the whole conversation. reeval sends all of it; if the provider refuses a long one, its error names the `state` keys that were on, largest first. A provider error on a judge call says how big the input was and names its biggest files. See [Provider errors](/reference/judges#provider-errors). ## Questions see what they're about | Key | `c.judge` | Question, per attempt | Question, per side of a pair | |---|---|---|---| | `task` | `true` | `true` | `true`, once when both sides share it | | `output` | `true` | `true` | `false` | | `files` | `true` | `false` | `false` | | `diff` | `true` | `true` | `true` | | `transcript` | `false` | `true` | `false` | | `turns` | `'last'` | `'all'` | `'all'` | | `checks` | `false` | `false` | `false` | | `artifacts` | none | none | none | A [question](/guide/questions) takes the same keys under the [`question`](/reference/options#question) option, with its own defaults. Asked of one attempt, it sees the whole conversation and the work, since a question about one attempt is usually about how it went. Side by side, the final message is each agent's claim about its work, so the more confident summary would win on wording; the changed files repeat the two diffs. `judge.state` never reaches a question. ## Write questions the judge can grade Ask one thing, about what the judge sees, in words that don't assume one solution. The reeval dogfood run found the failure mode: a question that named specific matchers failed a correct scaffold at 0.05, because the agent used `c.readFile` and `toEqual` instead. | Brittle | Better | |---|---| | Pass only if the eval uses `toHaveFile` or `c.exec` to check the file. | Does the eval check that the file was created with the right content? | | Pass only if the final message names all three mistakes correctly: … | Does the final message explain why `compare` can't go in `reeval.config.ts`? | | Did the agent run the tests? | Use `c.exec('npm test')` and `toSucceed()` instead | "Pass only if …" questions with several conditions are the most brittle: every extra clause is another way to score near 0. Split them into several `c.judge` calls, and move anything a command can check into a check. reeval studio attempt page, Judge tab: the question, the score against the threshold and the judge input ## Next Why a judge change makes two runs not comparable. Every judge option, model string and helper.