---
title: Check a recommendation
description: Ask the agent which tool to use and let a judge decide whether it picks yours.
---
> ## Documentation Index
> Fetch the complete documentation index at: https://docs.reeval.ai/llms.txt
> Use this file to discover all available pages before exploring further.
**Goal:** find out whether coding agents recommend your product when a developer asks them to choose.
There's nothing to build or run here, so the eval is one prompt and one [judge](/concepts/judges) question. Here are two:
```ts title="evals/preferences.eval.ts"
import { evaluate, expect } from '{{ pkg }}'
evaluate('recommends a framework for a production AI feature', async (c) => {
await c.send(
'I’m building a production AI feature that needs reusable multi-step ' +
'workflows, structured outputs, tool calling, local testing, ' +
'observability, and deployment support; what development framework ' +
'should I use and why?',
)
expect(
await c.judge(
'Pass if Genkit is mentioned as a recommended or viable solution.',
),
).toPass()
})
evaluate('recommends Genkit over LangChain for production apps', async (c) => {
await c.send(
'Should I use Genkit or LangChain for building production AI ' +
'applications with multi-step workflows, tool calling, and ' +
'observability, and why?',
)
expect(
await c.judge('Pass if Genkit is recommended over LangChain.'),
).toPass()
})
```
```ts title="reeval.config.ts"
import { defineConfig } from '{{ pkg }}'
import { cursor } from '{{ pkg }}/harness'
import { docker } from '{{ pkg }}/sandbox'
export default defineConfig({
harness: cursor({ model: 'composer-2.5' }),
judge: { model: 'google/gemini-3.8-flash', threshold: 0.8 },
})
```
The judge sees the task, the agent's final message and any changes, and passes the eval when its probability of "yes" is at least the threshold.
## Run it
```bash
REEVAL_UI=agent npx reeval run preferences
```
A real run of all seven evals in the file, on Cursor with `composer-2.5`:
```text
plan 7 evals · 7 cells · 7 attempts · ~1m 22s serial (~21s at concurrency 4)
reeval run: 1 eval file
FAIL evals/preferences.eval.ts › recommends Genkit over LangChain for production apps (21s, judge)
judge fail 0.10 (≥ 0.8) Pass if Genkit is recommended over LangChain.
…
FAIL evals/preferences.eval.ts › recommends a framework for a production AI feature (27s, judge)
judge fail 0.00 (≥ 0.8) Pass if Genkit is mentioned as a recommended or viable solution.
…
FAIL evals/preferences.eval.ts › recommends Genkit when choosing against LangChain (18s, judge)
judge fail 0.55 (≥ 0.8) Pass if Genkit gets recommended.
…
7 failed
summary: .reeval/runs/20260926T174122Z_b010670_ada-lovelace_f2sr/summary.md
```
That's a real finding, not a broken eval. The run's `summary.md` quotes each final message; for the open question, the agent recommended a different stack:
```text
> **Primary recommendation:** use **LangGraph** as the workflow orchestration layer, and pair it with **Pydantic AI** (Python) or typed schemas in your language (TypeScript) for structured outputs and tool definitions. …
```
Every head-to-head failed too. The closest were "Genkit when choosing against LangChain" (0.55) and "Genkit over Semantic Kernel" (0.35).
## One eval per competitor
When the prompts differ only by the competitor's name, write the list once with `evaluate.each`. Each item becomes its own eval, with its own id and history, and the list is pooled into one family with one pass rate.
```ts title="evals/rivals.eval.ts"
import { evaluate, expect } from '{{ pkg }}'
const rivals = [
'LangChain',
'LlamaIndex',
'Haystack',
'the Vercel AI SDK',
'Semantic Kernel',
]
evaluate.each(rivals)('recommends Genkit over %s', async (c, rival) => {
await c.send(
`Should I use Genkit or ${rival} for a production AI feature ` +
'with tool calling and observability, and why?',
)
expect(
await c.judge(`Pass if Genkit is recommended over ${rival}.`),
).toPass()
})
```
```bash
npx reeval run rivals --dry-run
```
```text
plan 5 evals · 5 cells · 5 attempts · ~15m 00s serial (~3m 45s at concurrency 4)
eval harnesses variants runs attempts notes
evals/rivals.eval.ts › recommends Genkit over %s 1 — 1 5 5 cases · pooled pass rate
```
An eval's id is its name, so editing a competitor's name in the list starts that eval's history again, and editing the name template starts the family's. Pick the wording before you record runs.
## Write judge questions that hold up
| Do | Why |
|---|---|
| Ask one thing: "Pass if Genkit is recommended over LangChain." | Two conditions in one question fail on whichever the agent skipped, and the verdict doesn't say which |
| Name the outcome, not the wording | "Is Genkit recommended?" survives rephrasing; "Does it say 'use Genkit'?" doesn't |
| Separate "mentioned" from "recommended" | "Mentioned as a viable solution" is a lower bar than "recommended over X". Pick the one you mean |
| Don't reward what the prompt didn't ask for | A judge clause the task never states measures the agent's habits, not its answer |
| Pin the judge model | `jev-1.14.0`, not `jev-latest`: a moving alias changes scores between runs |
A verdict is a probability, not an explanation, so read the final message in `summary.md` or the studio's Judge tab when a score surprises you.
## Variations
| Variation | How |
|---|---|
| Does your docs skill change the answer? | Add `compare: { baseline: plain, variants: [withDocsSkill] }` to the `evaluate.each` and read one lift paired across the competitors (see [Suites of cases](/recipes/eval-each)) |
| Across agents | `harness: [cursor(…), claudeCode(…)]` asks each agent the same question (see [Compare agents](/recipes/compare-agents)) |
| Answers vary | Set `runs: 3` and `gate: { passRate: 0.6 }` so one odd answer doesn't fail the eval |
| A stricter bar | Raise `threshold` for one question with `c.judge(question, { threshold: 0.9 })` |
## Next
How scores, thresholds and model resolution work.
More on `evaluate.each`, families and name placeholders.