--- title: Lift of a pull request description: >- Run a pull request's base and head trees as the two arms of one compare, and fail the check when the head doesn't help. --- > ## Documentation Index > Fetch the complete documentation index at: https://docs.reeval.ai/llms.txt > Use this file to discover all available pages before exploring further. **Goal:** for a pull request that changes what the agent reads (docs, skills, rules, templates), run the same eval on the base tree and on the head tree, side by side, and gate on the [lift](/concepts/lift). Both arms run in one run, under the same harness, model and image, so drift between days can't pass for a regression. It needs no recorded history. ## The files Each arm is an environment whose `workspace` is this repository at one commit. A missing variable throws, so an unset SHA can't silently run both arms on `HEAD`. ```ts title="evals/environments.ts" import { defineEnvironment } from '{{ pkg }}' function sha(name: string): string { const value = process.env[name] if (!value) throw new Error(`${name} isn't set: give it a commit SHA`) return value } const base = defineEnvironment('base', ({ git }) => ({ workspace: git('..', { ref: sha('BASE_SHA') }), })) const head = base.extendEnvironment('head', ({ git }) => ({ workspace: git('..', { ref: sha('HEAD_SHA') }), })) export const arms = { baseline: base, variants: [head] } ``` ```ts title="evals/greetings.eval.ts" import { evaluate, expect, group } from '{{ pkg }}' import { arms } from './environments.ts' const greetAda = `node -e "import('./greet.js').then((m) => { const out = m.greet('Ada') console.log(out) process.exit(out === 'Ahoy, Ada!' ? 0 : 1) })"` group( 'greetings guide', { compare: { ...arms, minLift: 0 }, runs: 4, gate: { passRate: 0.75 }, pin: ['compare', 'runs'], }, () => { evaluate('follows docs/greetings.md', async (c) => { await c.send( 'Read docs/greetings.md, then create greet.js exporting ' + 'greet(name) as it describes. No dependencies.', ) await expect(c.exec(greetAda), 'follows the guide').toSucceed() }) }, ) ``` | Option | What it does here | |---|---| | `git('..', { ref })` | The repository the eval file sits in, one level up from `evals/`, fetched on the host at that commit. A local path is relative to the calling file | | `extendEnvironment` | `head` shares everything with `base` and replaces only `workspace` | | `compare.minLift: 0` | Fails the head arm when the lift's 95% interval dips below 0 | | `runs: 4` | Four attempts a side: below that, the interval rarely clears 0 and effort can't reach significance | | `pin` | Keeps a `pr` profile's `limit` from turning `compare` off or cutting `runs` | ## Run it In this example, the pull request changes `docs/greetings.md` from `Hello, !` to `Ahoy, !`. ```bash export BASE_SHA=$(git rev-parse main) HEAD_SHA=$(git rev-parse HEAD) npx reeval run greetings ``` This is a real Cursor run on `composer-2.5`: ```text plan run 20261022T124219Z_… · 1 eval · 2 cells · 8 attempts · ~24m 00s serial (~6m 00s at concurrency 4) … pass greetings guide › follows docs/greetings.md arm pass base 0 of 4 head 4 of 4 helped check base head follows the guide 0 of 4 4 of 4 expected command to succeed, exit code 1 effort tools time tokens base 2 10s 60k head 8 19s 130k 45s · 789k in (373k cached) · 4,337 out · results .reeval/runs/20261022T124219Z_…/summary.md 1 pass ``` The base tree's guide says `Hello`, so its attempts fail the check. `reeval query lift` puts the lift at +100 points, with a 95% interval of +31 to +100. The effort rows show that the head did read the guide, at about twice the tokens. ## The workflow ```yaml title=".github/workflows/pr-lift.yml" name: pr-lift on: pull_request: paths: ['docs/**', 'skills/**', 'AGENTS.md', 'evals/**'] permissions: contents: read concurrency: group: pr-lift-${{ github.event.pull_request.number }} cancel-in-progress: true jobs: lift: # Forks get no secrets, so their PRs can't run real agents. if: github.event.pull_request.head.repo.full_name == github.repository runs-on: ubuntu-24.04 timeout-minutes: 45 env: BASE_SHA: ${{ github.event.pull_request.base.sha }} HEAD_SHA: ${{ github.event.pull_request.head.sha }} steps: - uses: actions/checkout@v7 with: fetch-depth: 0 persist-credentials: false - uses: actions/setup-node@v7 with: node-version: 22 cache: npm - run: npm ci - run: npx reeval image build - name: Run base and head env: CURSOR_API_KEY: ${{ secrets.CURSOR_API_KEY }} run: npx reeval run greetings - name: Upload runs if: ${{ !cancelled() }} uses: actions/upload-artifact@v7 with: name: reeval-runs-${{ github.run_id }} path: .reeval/runs include-hidden-files: true retention-days: 14 ``` | Step | Why | |---|---| | `paths` | Runs only when the pull request touches what the agent reads, or the evals | | Job-level `env` | Every `reeval` command loads the eval files, `reeval image build` included, so each needs both SHAs | | `fetch-depth: 0` | Actions checks out the merge commit alone. reeval fetches each SHA from this checkout on the host, and neither is in a shallow clone | | Run step | Exit 1 when the head arm falls short of `minLift`, which fails the check | `HEAD_SHA` is the pull request's own commit, not the merge commit Actions checks out. Use `${{ github.sha }}` instead to measure the merge result. For image caching, see the [PR gate](/recipes/github-pr-gate). ## When it goes wrong With the default `fetch-depth: 1`, preflight says what's missing and how to get it, and only the attempts on that tree fail (exit 3): ```text ✗ workspace ..#29111273208bd8e27c886d47681f59564284ea27 failed to prepare (set by environment base; only its attempts fail) Workspace git("..#29111273208bd8e27c886d47681f59564284ea27"): the repository has no ref "29111273208bd8e27c886d47681f59564284ea27" (a branch, tag or commit SHA). .. is a shallow clone, so it may not have that ref: fetch it (git fetch origin 29111273208bd8e27c886d47681f59564284ea27) or clone the full history (fetch-depth: 0 in actions/checkout). ``` Two arms that can only run the same tree are refused before anything runs (exit 2). Here, `BASE_SHA` and `HEAD_SHA` are the same commit: ```text reeval: evals/greetings.eval.ts: compare in evaluate("greetings guide › follows docs/greetings.md") runs head and base on the same tree: past what they share, each sets only workspace, and both are git("..") at ref "987de8c49be5ab0795f55fd1cfe5c343f96f7c2c". A ref read from an unset variable is undefined, so check the variables the arms take their refs from, and give each arm its own ref. ``` Different refs that resolve to the same tree, such as a pull request that changes nothing under a workspace's `path`, still run. The run prints a `warning:` line that names both refs. ## When to use which | | [PR gate](/recipes/github-pr-gate) | PR lift | |---|---|---| | Asks | Did an eval that passed on the base now fail? | Does the head tree help the agent more than the base? | | Base | The nearest recorded run on an ancestor commit | The base tree, in the same run | | History | Needed | Not needed | | A PR that edits an eval | `can't compare` | Runs the edited eval on both trees | | A PR that changes the config, judge or harness | Caught | Not caught: both arms get it | | Cost | One arm, once | Both arms, about four runs a side | The eval file always comes from the checkout, so on the base tree it may check for something the base never had. That's the point when the pull request changes only docs. When it changes code and its eval together, the base arm's failures say little. ## Variations | Variation | How | |---|---| | One package of a monorepo | `git('../..', { ref, path: 'packages/web' })`; see [Start from an existing codebase](/recipes/existing-codebase) | | Report only | `compare: { ...arms, gate: 'none' }` shows the lift without failing the check | | Keep the history | Runs in CI are recorded to `evals/.history/`; upload or commit them, and `reeval query lift` lists each pull request's lift | ## Next How the interval, `minLift` and effort work. Fail a pull request when an eval regresses.