--- title: Prove a skill helps description: >- Run one eval with and without a skill, gate on a pass rate and fail the run when the skill makes things worse. --- > ## Documentation Index > Fetch the complete documentation index at: https://docs.reeval.ai/llms.txt > Use this file to discover all available pages before exploring further. **Goal:** show that agents do better with your skill, and fail CI if it stops helping. You give the agent a skill through an environment, run the same eval with and without it, and read the difference as [lift](/concepts/lift). ## The files The skill teaches something the agent can't guess, so only the variant should pass. ```md title="skills/house-greeting/SKILL.md" --- name: house-greeting description: The house style for greetings in this organisation. Use when writing any greeting function, message or text. --- # House greeting style - The greeting word is always "Ahoy", never "Hello" or "Hi". - Format: `Ahoy, !` (comma after Ahoy, exclamation mark at the end). ``` ```ts title="evals/skill.eval.ts" import { defineEnvironment, evaluate, expect, group } from '{{ pkg }}' const plain = defineEnvironment('plain', {}) const withHouseGreeting = plain.extendEnvironment( 'with-house-greeting', ({ skill }) => ({ skills: [skill.local('../skills/house-greeting')] }), ) const greetAda = `node -e "import('./greet.js').then((m) => { const out = (m.greet ?? m.default)('Ada') console.log(out) process.exit(out === 'Ahoy, Ada!' ? 0 : 1) })"` group( 'house greeting', { compare: { baseline: plain, variants: [withHouseGreeting], minLift: 0 }, runs: 3, gate: { passRate: 0.6 }, pin: ['compare', 'runs'], }, () => { evaluate('greets in the house style', async (c) => { await c.send( 'Create greet.js exporting greet(name) that greets the person ' + 'in our house greeting style. Add a package.json with ' + '"type": "module". No dependencies.', ) await expect(c.exec(greetAda)).toSucceed() }) }, ) ``` ```ts title="reeval.config.ts" import { defineConfig } from '{{ pkg }}' import { cursor } from '{{ pkg }}/harness' import { docker } from '{{ pkg }}/sandbox' export default defineConfig({ harness: cursor({ model: 'composer-2.5' }), judge: { model: 'google/gemini-3.8-flash' }, }) ``` | Option | What it does here | |---|---| | `compare.variants` | Runs a variant cell with the skill next to the control, which has none | | `compare.minLift: 0` | Fails the variant when the lift's 95% interval dips below 0 | | `runs: 3` | Three attempts per cell, so the interval narrows | | `gate: { passRate: 0.6 }` | A cell passes at 2 of 3, so one flaky attempt doesn't fail CI | | `pin` | Keeps a `pr` profile's `limit` from turning `compare` off or cutting `runs` | `examples/with-skill-lift` in the reeval repository is a compare like this at four runs a side, the fewest that can say `helped`. Its skill says every change adds a line to `CHANGELOG.md`, and its config runs a fake agent that writes one only when it has the skill, so `pnpm eval` there shows a lift with no keys or Docker, and `--profile claude-code` runs it on Claude Code. ## Run it ```bash npx reeval run --dry-run npx reeval run skill ``` ```text plan 1 eval · 2 cells · 6 attempts · ~18m 00s serial (~4m 30s at concurrency 4) eval harnesses variants runs attempts notes evals/skill.eval.ts › house greeting › greets in the house style 1 1 3 6 ``` This is a real Cursor run of a house-greeting skill compare (`--runs 2`, no `minLift`): ```text pass house greeting › greets in the house style arm pass plain 0 of 2 with-house-greeting 2 of 2 can't tell yet check plain with-house-greeting house greeting 0 of 2 2 of 2 expected command to succeed, exit code 1 +2 more passed … 1m 31s · 5,383k in (2,593k cached) · 12k out · results .reeval/runs/20261020T213308Z_…/summary.md 1 pass ``` Without the skill, the control attempts wrote greetings such as `Welcome home, Ada!`. With it, both variant attempts wrote `Ahoy, Ada!`. ## Read the lift The recipe as written, under the [fake harness](/recipes/fake-harness) (which writes the house greeting only when it has a skill), with `-v`: ```text pass house greeting › greets in the house style arm pass plain 0 of 3 with-house-greeting 3 of 3 can't tell yet difference +100 points, 95% interval +21…+100 check plain with-house-greeting toSucceed node -e "import('./greet.js').then((m… 0 of 3 3 of 3 expected command to succeed, exit code 1 ``` - Each arm's count is its passing tries out of those that counted: `0 of 3` and `3 of 3`. - The word after the variant's count is its effect: `helped`, `made it worse` or `can't tell yet`. Below four tries a side it's always `can't tell yet`. - `-v` adds the difference in points and its 95% interval. The interval's lower bound is above `minLift` (0), so the eval passes. - The control has no gate, so its misses don't change the exit code. - The check table shows which check told the arms apart. An unlabelled check is named by its matcher and argument; label it with `expect`'s second argument. ## Read the effort ```bash npx reeval run skill --runs 4 npx reeval query effort npx reeval analyze ``` A skill can help without changing the pass rate: the agent gets there with less searching. At four tries a side, the block adds an effort table of each arm's medians. A real Cursor run: ```text pass house greeting › greets in the house style arm pass plain 0 of 4 with-house-greeting 4 of 4 helped check plain with-house-greeting house greeting 0 of 4 4 of 4 expected command to succeed, exit code 1 +2 more passed effort tools time tokens plain 54 1m 39s 2,890k with-house-greeting 3 6.6s 57k … 2m 00s · 13,147k in (6,400k cached) · 28k out · results .reeval/runs/20261020T214734Z_…/summary.md 1 pass ``` The variant needed 3 tool calls against the control's 54. Behind the table is a rank test on tool calls, agent time and tokens (when both sides reported usage); `reeval query effort` has its p-values and verdicts (`fewer`, `more`, `noise`, `insufficient`). [Effort](/concepts/lift#effort-next-to-lift) has the details. `reeval query effort` lists the same numbers for every recorded run. [`reeval analyze`](/reference/cli/analyze) explains a run in plain words, lift and effort included, with each claim tied to a fact from the run. ## When the skill doesn't help The same eval, with an agent that already writes `Ahoy` without the skill, with `-v`: ```text fail house greeting › greets in the house style arm pass plain 3 of 3 with-house-greeting 3 of 3 can't tell yet, not by minLift 0 difference +0 points, 95% interval −56…+56 836ms · results .reeval/runs/20261020T214748Z_…/summary.md 1 fail ``` Every attempt passed, and the run still exits 1: `minLift: 0` asks for confidence that the skill is no worse, and three attempts per side can't give it. `minLift` checks the interval's lower bound, not the lift. At `runs: 3`, only a large effect clears 0, and a control that already passes never does. Use `minLift` on evals the agent fails without help, or raise `runs`. ## Variations | Variation | How | |---|---| | Several skills | `compare: { baseline: plain, variants: [good, weak], minLift: 0 }` gives each its own cell and lift against the same control | | Skill from git or npm | `skill.git('github:org/skills', { ref: 'v2.4.0', path: 'skills' })` or `skill.npm('pkg@1.2.3')`. Pin the ref: a moving one warns in the plan | | A shared starting point | `const withSkill = base.extendEnvironment('with-skill', …)` then `compare: { baseline: base, variants: [withSkill] }`. `compare` does not layer them | | Report only | Drop `minLift` and set `compare: { baseline, variants, gate: 'none' }` to see lift without failing anything | | Ask which arm did better | `questions: [{ question: 'Which greet.js reads more like our house style?', across: 'environment' }]` beside `compare` judges the two arms' diffs pair by pair after the run. Only attempts that pass their checks are paired, so here, where the control is expected to fail, ask each attempt instead: `questions: ['How did the agent pick its greeting?']`; see [Questions](/guide/questions) | | Across runs | `reeval query lift` lists every variant cell's lift from recorded runs (see [Analysis](/guide/analysis)) | | Did the agent open the skill? | `expect(turn).toHaveCalledSkill('house-greeting')` after `c.send`. That is "opened or loaded", not "followed". Keep the `c.exec` check for the greeting. See [Matchers](/reference/matchers#transcript-matchers) | ## Next How the interval, `minLift` and "within noise" work. The same compare, with an MCP server as the variant.