--- title: CI description: >- Gate pull requests on your evals in GitHub Actions, with Docker on Ubuntu runners. --- > ## Documentation Index > Fetch the complete documentation index at: https://docs.reeval.ai/llms.txt > Use this file to discover all available pages before exploring further. In CI, `reeval run` needs no extra flags. When `CI` is set it prints plain, grouped output, picks the `pr` profile and records the run. ```bash CI=1 npx reeval run hello jq --output-logs full ``` ```text plan run 20261020T213201Z_a3fbded_ci_yqkc · profile pr (auto: CI is set) · 3 evals · 3 cells · 3 attempts · ~39s serial (~13s at concurrency 4) · budget 20 attempts ✓ … reeval 0.10.0 · run 20261020T213201Z_a3fbded_ci_yqkc · 2 eval files · harness cursor:composer-2.5 · sandbox docker · concurrency 4 · grouped logs [summarises#1 cursor] ▶ evals/jq.eval.ts › summarises a JSON file with jq [summarises#1 cursor] +0:00 sandbox ready 2s [summarises#1 cursor] +0:02 turn 1 done 6s · 4 tool calls, 2 file edits [summarises#1 cursor] +0:08 $ jq --version exit 0 48ms [summarises#1 cursor] +0:08 $ ./count.sh exit 0 57ms … pass greet module › creates a tested greet module pass greet module › adds a default name in a follow-up turn pass summarises a JSON file with jq 20s · 211k in (87k cached) · 1,896 out · results .reeval/runs/20261020T213201Z_a3fbded_ci_yqkc/summary.md 3 pass Set analysis.model in reeval.config.ts for answers to what each eval asks. recorded evals/.history/20261020T213201Z_a3fbded_ci_yqkc.json + 14 artifact files in evals/.history/20261020T213201Z_a3fbded_ci_yqkc/ (392.9 KB). Commit it with git. ``` ## What CI changes | Setting | In CI | Elsewhere | |---|---|---| | Output (`--ui`) | `plain`: append-only lines, then the result blocks | `live` on a terminal, `agent` under a coding agent | | Log order | `grouped`: each attempt as one block when it ends | `stream` | | Profile | `defaultProfile.ci`, here `pr` | `defaultProfile.local`, or none | | Recording | `record: 'ci'` records every run | Only with `reeval record` or `--record` | | Author in run ids | `ci` | Your git user | After 60 seconds without output, plain mode prints what's still running and in which phase, so CI doesn't kill a quiet job. `REEVAL_HEARTBEAT=30s` changes the interval, and `0` turns it off. [Output modes](/concepts/output) has the details. ## Output on GitHub Actions ```text ::group::[creates#1 cursor] ▶ evals/hello.eval.ts › greet module › creates a tested greet module [creates#1 cursor] +0:00 sandbox ready 2s [creates#1 cursor] +0:01 turn 1 done 5s · 1 tool call, 2 file edits [creates#1 cursor] +0:06 $ node --test exit 0 162ms [creates#1 cursor] +0:07 judge pass 1.00 (≥ 0.8) Does the agent's final message accurately describe what it changed? ::endgroup:: ``` When `GITHUB_ACTIONS` is set, failing attempts stay open and add an `::error` annotation on the failing line. By default a passing attempt prints nothing; with `--output-logs full`, as in this capture, it folds into a `::group::` block titled with its start line. Lines inside a group drop the `[creates#1 cursor]` prefix unless you pass `--log-prefix task`, as this capture did. The result blocks follow, outside any group. reeval also appends the run's `summary.md` to the job summary (`$GITHUB_STEP_SUMMARY`), so the result is on the run's page. At the start of `reeval run` it writes `run-id=` to `$GITHUB_OUTPUT` and `run ` to the step summary, so the job log has the id even when the runner is killed before assemble. To continue that run after a timeout, restore `.reeval/staging` (or the interrupted scratch) onto the new VM and run `reeval run --resume`, or `reeval run --resume ` with the value from `run-id`. Without those files, `--resume` has nothing to continue. See [Resume a stopped run](/reference/cli/run#resume-a-stopped-run). ## Gate pull requests ```yaml title=".github/workflows/evals.yml" name: evals on: pull_request: permissions: contents: read jobs: evals: runs-on: ubuntu-24.04 timeout-minutes: 30 steps: - uses: actions/checkout@v7 with: persist-credentials: false fetch-depth: 0 - uses: actions/setup-node@v7 with: node-version: 22 cache: npm - run: npm ci - run: npx reeval image build - name: evals env: CURSOR_API_KEY: ${{ secrets.CURSOR_API_KEY }} GOOGLE_GENERATIVE_AI_API_KEY: ${{ secrets.GOOGLE_GENERATIVE_AI_API_KEY }} run: npx reeval run - name: compare with the base branch if: ${{ !cancelled() }} run: | npx reeval compare --md >> "$GITHUB_STEP_SUMMARY" npx reeval compare --fail-on-regression - if: ${{ !cancelled() }} uses: actions/upload-artifact@v7 with: name: reeval-runs path: .reeval/runs include-hidden-files: true retention-days: 14 ``` Ubuntu runners have Docker running, so `reeval image build` and `reeval run` work as they do locally, and `reeval run` fails the job when a gated cell fails. `reeval compare` then diffs the run against the nearest recorded run in the branch's git history, and `--md` writes the evals that changed to the job summary: each regressed eval with the checks that flipped, and a tally such as `1 regressed · 4 same`. `--fail-on-regression` fails the job if any eval regressed, or a new or changed eval fails. With no recorded run to compare with, every eval is new. `fetch-depth: 0` gives `reeval compare` the git history it needs to find that ancestor. In a shallow clone it falls back to the previous run by time. ## Keep keys out of reach ```yaml - uses: actions/checkout@v7 with: persist-credentials: false fetch-depth: 0 # … - run: npx reeval run env: CURSOR_API_KEY: ${{ secrets.CURSOR_API_KEY }} ``` Add `CURSOR_API_KEY`, your judge's key and `analysis.model`'s key as repository secrets, and pass them only to the step that runs `reeval`. Eval files run on the runner with those keys, so `persist-credentials: false` keeps a git token out of the same job. GitHub doesn't give secrets to pull requests from forks, so a fork's eval code never runs with your keys; its run stops on the missing key instead. ## Insights and remarks in CI ```text pass creates greet.js insights skipped: set GOOGLE_GENERATIVE_AI_API_KEY, or analysis: { insights: false } 11s · 63k in (22k cached) · 386 out · results .reeval/runs/…/summary.md 1 pass ``` A CI run writes [insights](/concepts/insights) as a local one does, and `summary.md` takes them and the remarks' concerns into the job summary. Without `analysis.model`'s key, the run prints this line and its facts, with the same exit code. `--no-insights` skips them in a job that shouldn't spend tokens. Remarks need no key. `check-never-fails` counts runs recorded on the default branch: `origin`'s HEAD, a local `main` or `master`, else `GITHUB_BASE_REF`, which is what a GitHub Actions pull request build has. ## Record runs from CI ```yaml - run: npx reeval run --profile nightly - run: | git config user.name "github-actions[bot]" git config user.email "github-actions[bot]@users.noreply.github.com" git add evals/.history git commit -m "Record nightly eval run" git push ``` A PR run is recorded on the runner, in `evals/.history/`, but nothing commits it, so the upload step keeps it. To build a history, record from a scheduled job on your main branch and commit the result. That job needs `contents: write` and a checkout that keeps its credentials. [Nightly matrix and Pages](/recipes/nightly-pages) has the complete workflow. ## Next Ask questions across runs with SQL. The complete PR gate recipe.