--- title: Gates and lift description: Which cells can fail a run, how a cell passes, and how sure a lift is. --- > ## Documentation Index > Fetch the complete documentation index at: https://docs.reeval.ai/llms.txt > Use this file to discover all available pages before exploring further. A `compare` eval answers "does this environment help?". Two settings decide what fails the run, and lift, with its interval, says how big the difference is and whether the attempts can tell. ```ts title="evals/skill.eval.ts" group('house greeting', { compare: { baseline: plain, variants: [withHouseGreeting], minLift: 0 }, gate: { passRate: 0.8 }, runs: 5, }, () => { evaluate('greets in the house style', async (c) => { await c.send('Create greet.js exporting greet(name) in our house style.') await expect(c.exec(greetAda)).toSucceed() }) }) ``` Run under the fake agent, which writes the house greeting only when it has the skill: ```text pass house greeting › greets in the house style arm pass plain 0 of 5 with-house-greeting 5 of 5 helped check plain with-house-greeting house greeting 0 of 5 5 of 5 expected command to succeed, exit code 1 … ``` ## Choose which cells gate `compare.gate` picks which cells can fail the run. The control usually runs without the thing you're testing, so it's expected to do worse. | `compare.gate` | Variant cells | Control cell | |---|---|---| | `'variants'` (default) | Gate | Report only: its misses print, but never fail the eval | | `'all'` | Gate | Gate | | `'none'` | Report only | Report only | A cell that doesn't gate has `gate: 'none'` in the plan and the record. Without `compare`, the one cell always gates, and so does a control a profile's `limit.compare` leaves alone. ## Choose how a cell passes The `gate` option (on the config, a profile's `defaults`, a group or an eval) decides when a gating cell passes. | `gate` | A cell passes when | Elsewhere called | |---|---|---| | `'all'` (default) | Every counted attempt passes | pass^k | | `{ passRate: 0.8 }` | Its pass@1 is at least 0.8 | a pass@1 threshold | A cell's pass@1 is its passes over its passes and fails: the chance that one attempt passes. Infra errors and skipped attempts don't count either way. `'all'` gets harder to meet as `runs` grows, even for an agent that's no worse: at a pass@1 of 0.85, all five of `runs: 5` pass only 0.85⁵ = 44% of the time. The plan prints a hint when an eval runs more than once with it. ## Lift and its interval Lift is the variant's pass@1 minus the control's on the same harness, in points. reeval stores it on the variant cell with a 95% interval on the difference, computed with Newcombe's method from each side's Wilson score interval. ```mermaid flowchart LR C["control: 0/5 → 0%"] --> L["lift = 100% − 0% = +100 pts"] V["variant: 5/5 → 100%"] --> L L --> I["95% interval +39…+100"] ``` The arm's effect word comes from the interval: `helped` when it's above 0, `made it worse` when it's below, `can't tell yet` when it spans 0, and blank when the two sides passed alike. Below four counted attempts a side it's always `can't tell yet`. `-v` prints the interval under the arm: ```text with-house-greeting 5 of 5 helped difference +100 points, 95% interval +39…+100 ``` The Wilson interval stays sensible at 0/n and n/n, which is where small evals usually land. Lift gates; [questions](/guide/questions) and their win rates only annotate, and never change a cell's outcome. Each judge key also has its own pass rate on the cell (`login 4/5 · checkout 0/5`) in the studio. A key's lift shows only when both sides have at least four counted attempts, the same floor as the arm's effect. ## Why two runs is usually noise The interval is wide when attempts are few. These are reeval's own intervals for common results: | Variant | Control | Lift | 95% interval | Within noise | |---|---|---|---|---| | 1/1 | 0/1 | +100 | −12…+100 | yes | | 2/2 | 2/2 | +0 | −66…+66 | yes | | 2/2 | 1/2 | +50 | −27…+91 | yes | | 2/2 | 0/2 | +100 | +7…+100 | no | | 5/5 | 3/5 | +40 | −12…+77 | yes | | 5/5 | 0/5 | +100 | +39…+100 | no | | 10/10 | 5/10 | +50 | +12…+76 | no | | 20/20 | 10/20 | +50 | +24…+70 | no | At two runs per cell only an all-or-nothing difference clears noise, and reeval still says `can't tell yet` there: two tries a side is too few to call. Every view uses the same floor of four counted attempts a side, so `reeval query factor_effects`, `contrasts`, a judge key's lift and `reeval analyze` say `insufficient` where the run says `can't tell yet`. The same eval at `--runs 2 -v`: ```text pass house greeting › greets in the house style arm pass plain 0 of 2 with-house-greeting 2 of 2 can't tell yet difference +100 points, 95% interval +7…+100 … ``` Four a side keeps the views agreeing; it doesn't make the stated 95% exact. Below about ten attempts a side, an `effect` can be a false positive more often than 1 time in 20: at 4 against 4, a 4/4 against 0/4 split happens by chance about 7% of the time when nothing changed. Lift only measures pass@1. A skill can also show up as fewer tool calls or less time, which the [effort table](#effort-next-to-lift) reports. Use `reeval query lift` to see every variant cell's lift across runs, with `lo`, `hi` and `within_noise` columns. ## Lift across cases An [`evaluate.each`](/recipes/eval-each) family with `compare` gets one lift per variant, paired by case: each case ran the control and the variant, so only the cases where they disagree (discordant) move it. ```text greets %s in the house style pass Ada arm pass plain 0 of 1 with-house-greeting 1 of 1 … pass all cases arm pass plain 0 of 6 with-house-greeting 6 of 6 helped with-house-greeting: 6 paired cases, 95% interval +45…+100 ``` The `all cases` row pools the cases into one effect; `-v` adds its paired interval. `reeval query lift` has the p-value. | Runs per cell | Interval | p | `insufficient` below | |---|---|---|---| | 1 | Newcombe's paired score interval (his method 10) | McNemar's exact test | 6 discordant cases | | More | A t interval over each case's pass-rate difference | Wilcoxon signed-rank | 6 cases | Six discordant cases are the fewest whose exact p can fall below 0.05 (2 · 0.5⁶ = 0.031). The verdict is `lift` or `drop` when the interval excludes 0, else `noise`, and a family's effort uses the same per-case pairing with a signed-rank verdict. ## Effort next to lift Pass rate can't tell a skill that makes the agent four times cheaper from no skill at all when both pass. So a measure-a-change block ends with an effort table: each arm's median tool calls, agent time and, when the harness reports them, tokens. It needs four counted attempts a side, and shows when the eval failed or an arm's effort differs from its baseline's by the rank test below. From a run of three arms at four tries each: ```text fail greets in the house style arm pass plain 0 of 4 with-skill 4 of 4 helped with-doc 0 of 4 … effort tools time plain 1 <0.1s with-skill 1 <0.1s with-doc 1 <0.1s ``` The medians cover passes and fails; infra errors and skips don't count. The verdict comes from an exact, two-sided Mann–Whitney rank test on tool calls, the steadier measure, so `fewer` and `more` say which arm's attempts rank lower: equal medians can still be `fewer`. Agent time and tokens get their own verdicts in `reeval query effort`, and effort never fails a run. Tokens join the table when both sides reported usage. An attempt with no usage (Goose's, say) drops out of the token sample only, and when either side has none left the token metric is omitted, not `insufficient`. **Effort** always means this measure, an output read from the attempt. A harness factory's `effort` option, as in `codex({ effort: 'high' })`, is **reasoning effort**, an input to the model, which `reeval query` has as `reasoning_effort`. | Verdict | Printed | When | |---|---|---| | `fewer`, `more` | `fewer tool calls (p = 0.029)` | p is below 0.05 | | `noise` | `within noise` | p is 0.05 or more | | `insufficient` | `too few attempts to test` | No split of these values could reach p below 0.05 | Without ties, the smallest p a rank test can give depends only on the attempt counts, so a verdict needs at least four attempts on each side. Tied values (tool calls often are) move that floor either way: `2, 2, 2, 3` against `2, 2, 3, 3` is `insufficient` at four a side, while `1, 2, 3` against `4, 4, 4, 4` is `fewer` at p = 0.029. Equal sides still need four: | Variant vs control | Smallest p without ties | Can reach a verdict | |---|---|---| | 2 vs 2 | 0.333 | no | | 3 vs 3 | 0.100 | no | | 3 vs 4 | 0.057 | only with ties | | 4 vs 4 | 0.029 | yes, unless ties prevent it | | 5 vs 5 | 0.008 | yes, unless ties prevent it | So `reeval run` hints when a compare runs 2 or 3 times a cell: `hint: 1 compare runs 3 times per cell; effort can't reach p < 0.05 below 4 vs 4 (runs: 4)`. ### No overlap at small n At three or more attempts a side, a metric whose two ranges don't overlap is marked `separated` (`time_separated`, `tokens_separated`) in `reeval query effort`, even when its test is `insufficient`. It's a direction, not an effect: one ordering in ten separates by chance at 3 vs 3. ```bash reeval query "SELECT eval_key, tools_median, separated FROM effort WHERE role = 'variant'" ``` ## Effects across runs ```text eval_key factor tier points lo hi pass_verdict effort_verdict evals/skill.eval.ts › house greeting › greets in the house… environment controlled 1 0.07 1 insufficient insufficient … evals/skill.eval.ts › house greeting › greets in the house… environment pooled 1 0.307 1 effect fewer · harness pooled 0 -0.436 0.436 noise insufficient ``` `reeval query factor_effects` contrasts one factor at a time (`environment`, `harness` or `model`), with a pass-rate verdict from the Newcombe interval and an effort verdict from the rank test. Pooling is how two runs at a time add up to a verdict: each run's skill contrast is `insufficient` on pass rate and effort, and the pooled one isn't. | `tier` | The contrast | |---|---| | `controlled` | Two cells in one run and eval | | `pooled` | The same contrast over comparable runs: attempts summed per eval, then across evals, where an effort effect needs the same verdict in at least 2/3 of them. Two source fingerprints of the same eval are not pooled | | `paired` | A variant against its control over a pooled `evaluate.each` family's cases in one run; `strata` is the paired cases | | `observed` | A cell between two runs whose fingerprint changed in one part; `confounded` when several did | | `not_measured` | The factor has one level | [`reeval analyze`](/guide/analyze) reads these facts and explains a run in plain words. ## Gate on the lower bound with minLift `minLift` (points from -1 to 1) fails a variant cell whose interval's **lower bound** is below it, so it asks for confidence, not a lucky point estimate. `minLift: 0` means "confidently no worse than the control". With `minLift: 0.5` at `runs: 2`, a perfect +100 still fails: ```text fail house greeting › greets in the house style arm pass plain 0 of 2 with-house-greeting 2 of 2 can't tell yet, not by minLift 50 … 1 fail ``` A variant below its `minLift` fails the run with exit 1: add runs until the interval narrows. On an `evaluate.each` family, `minLift` gates the pooled lift's lower bound, not each case's, so more cases narrow it too. ## Methods reeval uses standard methods for small samples, so its numbers can be checked against any statistics package. | Method | Used for | Source | |---|---|---| | Wilson score interval | Each side's pass@1 | Wilson, E. B. (1927). *Probable inference, the law of succession, and statistical inference.* JASA 22(158) | | Newcombe's hybrid score interval | The 95% interval on lift | Newcombe, R. G. (1998). *Interval estimation for the difference between independent proportions.* Statistics in Medicine 17(8) | | Exact two-sided Mann–Whitney U test | The effort verdict | Mann, H. B. and Whitney, D. R. (1947). *On a test of whether one of two random variables is stochastically larger than the other.* Annals of Mathematical Statistics 18(1) | ## See it in the studio The run page shows each cell's role, pass rate and lift with its interval. reeval studio run page with a cells table: the control at 0%, the variant at 100% and its lift ## Next How a judge verdict becomes a pass or a fail. A complete compare eval with a pass-rate gate and `minLift`.