--- title: PR gate in GitHub Actions description: >- Run your evals on every pull request, post the results to the job summary and fail the check when an eval regresses. --- > ## Documentation Index > Fetch the complete documentation index at: https://docs.reeval.ai/llms.txt > Use this file to discover all available pages before exploring further. **Goal:** every pull request that touches evals, skills or the config runs the real agent in Docker, and the check fails when an eval that passed on the base branch now fails. This workflow passes [actionlint](https://github.com/rhysd/actionlint), and each `reeval` command in it was run locally, but the workflow as a whole hasn't run in GitHub Actions. Check the action versions against your organisation's pins before you rely on it. ## The files A `pr` profile keeps pull request runs cheap: one attempt per cell and no variants. `defaultProfile` picks it whenever `CI` is set. ```ts title="reeval.config.ts" import { defineConfig } from '{{ pkg }}' import { cursor } from '{{ pkg }}/harness' import { docker } from '{{ pkg }}/sandbox' export default defineConfig({ harness: cursor({ model: 'composer-2.5' }), judge: { model: 'google/gemini-3.8-flash' }, defaultProfile: { ci: 'pr' }, profiles: { pr: { limit: { runs: 1, compare: false }, budget: { attempts: 20 }, }, }, }) ``` ```yaml title=".github/workflows/evals.yml" name: evals on: pull_request: paths: ['evals/**', 'skills/**', 'reeval.config.ts', 'package-lock.json'] permissions: contents: read concurrency: group: evals-${{ github.event.pull_request.number }} cancel-in-progress: true jobs: evals: # Forks get no secrets, so their PRs can't run real agents. if: github.event.pull_request.head.repo.full_name == github.repository runs-on: ubuntu-24.04 timeout-minutes: 45 steps: # Eval code runs on the host with the keys, so the job keeps no git token. - uses: actions/checkout@v7 with: fetch-depth: 0 persist-credentials: false - uses: actions/setup-node@v7 with: node-version: 22 cache: npm - run: npm ci - name: Restore sandbox images id: images uses: actions/cache@v6 with: path: ${{ runner.temp }}/reeval-images.tar key: >- reeval-images-${{ hashFiles('package-lock.json', 'reeval.config.ts', 'docker/**') }} - name: Load sandbox images if: steps.images.outputs.cache-hit == 'true' run: docker load -i "$RUNNER_TEMP/reeval-images.tar" - name: Build sandbox images if: steps.images.outputs.cache-hit != 'true' run: | npx reeval image build --profile pr images=$(docker images --format '{{.Repository}}:{{.Tag}}' \ | grep '^reeval-') docker save -o "$RUNNER_TEMP/reeval-images.tar" $images - name: Run evals env: CI: 1 CURSOR_API_KEY: ${{ secrets.CURSOR_API_KEY }} GOOGLE_GENERATIVE_AI_API_KEY: ${{ secrets.GOOGLE_AI_KEY }} run: | set +e npx reeval run --profile pr code=$? # 1 is a failing eval: the compare decides. 2 and 3 are errors. if [ "$code" -gt 1 ]; then exit "$code"; fi - name: Compare with the base branch run: | set +e npx reeval compare --md --fail-on-regression > compare.md code=$? cat compare.md >> "$GITHUB_STEP_SUMMARY" exit "$code" - name: Upload runs if: ${{ !cancelled() }} uses: actions/upload-artifact@v7 with: name: reeval-runs-${{ github.run_id }} path: .reeval/runs include-hidden-files: true retention-days: 14 ``` ## What each step does | Step | Why | |---|---| | `if:` on the job | A fork's pull request gets no secrets, so the job is skipped rather than failing on missing keys | | `persist-credentials: false` | Eval files are code that runs on the host next to your keys; the job keeps no git token | | `fetch-depth: 0` | `reeval compare` walks the commit's ancestors to find the base run; a shallow clone has none | | Image cache | `reeval image build` bakes the agent's CLI into reeval's image. The tag follows the installed package, so the lockfile is in the key. `install` and `dockerfile` images that change later build on first use | | Run step | Exit 1 (an eval failed) doesn't end the job, so a failure that already failed on the base doesn't block the PR. Exit 2 (config) and 3 (infra) do | | Compare step | Writes the Markdown table to the job summary and exits 1 on a regression | Add `CURSOR_API_KEY` and `GOOGLE_AI_KEY` under **Settings › Secrets and variables › Actions**. Keys never enter the sandbox; see [Secrets and egress](/guide/secrets-and-egress). ## What the run prints Under GitHub Actions, `reeval run` keeps a failing attempt's detail open under an `::error` and appends its own `summary.md` to the job summary; with `--output-logs full` it also wraps each passing attempt in a collapsible group titled with its start line. This is a real Cursor run of a project of greeting evals with `GITHUB_ACTIONS=true CI=1` and `--output-logs full`: ```text plan run 20261020T213231Z_… · profile pr (auto: CI is set) · 2 evals · 2 cells · 2 attempts · ~26s serial (~13s at concurrency 4) · budget 20 attempts ✓ preparing 1 sandbox: docker reeval 0.10.0 · run 20261020T213231Z_… · 1 eval file · harness cursor:composer-2.5 · sandbox docker · concurrency 4 · grouped logs ::group::[creates#1 cursor] ▶ evals/hello.eval.ts › greet module › creates a tested greet module [creates#1 cursor] +0:00 sandbox ready 2s [creates#1 cursor] +0:01 turn 1 done 5s · 1 tool call, 2 file edits [creates#1 cursor] +0:06 $ node --test exit 0 162ms [creates#1 cursor] +0:07 judge pass 1.00 (≥ 0.8) Does the agent's final message accurately describe what it changed? ::endgroup:: … pass greet module › creates a tested greet module pass greet module › adds a default name in a follow-up turn 14s · 141k in (61k cached) · 840 out · results .reeval/runs/20261020T213231Z_…/summary.md 2 pass Set analysis.model in reeval.config.ts for answers to what each eval asks. recorded evals/.history/20261020T213231Z_….json + 10 artifact files in evals/.history/20261020T213231Z_…/ (376.4 KB). Commit it with git. ``` Because `CI` is set, the run is recorded into `evals/.history` on the runner. Nothing commits it from a pull request; the upload step keeps `.reeval/runs` for 14 days. ## What a regression looks like This output came from two recorded runs of a fake-harness project, not a real agent, where an environment variable made one check fail on the second run: ```bash npx reeval compare --md --fail-on-regression ``` ````text ### reeval compare: 20261020T231408Z_b2c98b4_ada-lovelace_fypq vs 20261020T231407Z_ea2a649_ada-lovelace_8h1n ```text … regressed greets in the house style base 20261020T231407Z_ea2a649_ada-lovelace_8h1n head 20261020T231408Z_b2c98b4_ada-lovelace_fypq house greeting pass → fail expected command to succeed, exit code 1 sign-off not run +1 more passed +2 same … 1 regressed · 1 fixed · 1 can't compare · 1 new · 1 removed · 4 same ``` ```` The command exits 1, so the step and the check fail. In the job summary the blocks render as they print in a terminal. A `regressed` eval fails the check, and so does a `new` or `can't compare` eval that fails; a changed eval that passes doesn't. ## The first run `reeval compare` compares with a recorded run on an ancestor of the PR's commit. Until one exists, every eval is new, and `--fail-on-regression` fails the check only if one of them fails: ```text No earlier recorded run to compare 20261020T231404Z_ea2a649_ada-lovelace_gtej with, so every eval is new. pass bills in minor units with the billing doc … 8 new evals (3 fail · 5 pass) ``` To give pull requests a base, commit runs to `evals/.history` on your main branch; [Nightly matrix and Pages](/recipes/nightly-pages) does it every night. ## Variations | Variation | How | |---|---| | Gate on the run too | Drop the `set +e` lines so any failing eval fails the check, whatever the base did | | Only some evals | `npx reeval run --profile pr --tag smoke` or a filter such as `npx reeval run hello`. `reeval image build` takes the same path and name filters (`npx reeval image build hello`), not `--tag` | | More harnesses | Add their keys as secrets and env entries; reeval checks only the keys of the harnesses the profile keeps | | A fixed base | `npx reeval compare latest --md` compares with a named run instead of the nearest ancestor | | Pin actions by SHA | Reeval's own workflows pin `actions/checkout`, `actions/cache` and `actions/upload-artifact` to commit SHAs with the version in a comment | ## Next Record a base every night and publish the studio. How the base is chosen and what each change means.