---
title: Run evaluations in CI
description: Fail a CI job when a recorded run regresses.
---
> ## Documentation Index
> Fetch the complete documentation index at: https://docs.reeval.ai/llms.txt
> Use this file to discover all available pages before exploring further.

Run evaluations on each pull request opened from this repository so the job keeps a recorded run and fails when an evaluation regresses. Create a profile named `pr` before adding the workflow. For more information about profiles, see [Profiles](/evaluations/runs/profiles).

## Add the workflow

<Steps>
<Step title="Run evaluations">

Add the `.github/workflows/reeval.yml` file. It runs the evaluations on each pull request opened from this repository.

```yaml title=".github/workflows/reeval.yml"
name: Evaluations
on:
  pull_request:
jobs:
  reeval:
    if: >-
      ${{ github.event.pull_request.head.repo.full_name ==
      github.repository }}
    runs-on: ubuntu-latest
    permissions:
      contents: write
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0
          ref: ${{ github.head_ref }}
      - uses: actions/setup-node@v4
        with:
          node-version: {{ node }}
      - run: npm ci
      - id: reeval
        env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
        run: |
          set +e
          npx reeval run --profile pr
          code=$?
          if [ "$code" -gt 1 ]; then exit "$code"; fi
```

The `ubuntu-latest` runner includes the container runtime the sandbox uses. `fetch-depth: 0` keeps every ancestor commit, so `reeval compare` can pick a base run from one of them. The `ref` input checks out the pull request branch, and the `contents: write` permission lets a later step push to that branch.

The run step's id is `reeval`. At the start of the run, the `reeval run` command writes `run-id=<id>` to `GITHUB_OUTPUT` and appends `run <id>` to `GITHUB_STEP_SUMMARY`.

When the run finishes, the `reeval run` command appends the run's `summary.md` file to the job summary. A failed run receives that file too. With `GITHUB_ACTIONS` set, a failing attempt stays expanded in the log and adds an error annotation on its evaluation file.

The `reeval run` command exits 1 when an evaluation fails. The step lets that code through so the later `reeval compare` step decides, because an evaluation that already failed on the base commit is not a regression. Any other non-zero code, such as 2 for a usage error or 3 for an infrastructure error, fails the job here.

An ungated cell, a baseline by default, can fail on a run that exits 0. A `budget.duration` stop counts as a failed run. For more information about exit codes, see [Exit codes](/reference/cli#exit-codes).

</Step>
<Step title="Add the API key">

Add the harness's API key to the repository secret used by the run step.

</Step>
<Step title="Record results">

In the `.github/workflows/reeval.yml` file, add this step to commit the history directory and push it, so a later run can use this one as its base.

```yaml title=".github/workflows/reeval.yml"
      - name: Commit the history directory
        if: ${{ !cancelled() }}
        run: |
          git config user.name "github-actions[bot]"
          git config user.email "reeval-ci@example.com"
          git add evals/.history
          git commit -m "chore: record reeval run"
          git push
```

With the default `record: 'ci'`, reeval copies each finished run into `evals/.history` when the `CI` variable is set ([the `record` option](/reference/options#record)). The copy includes a run whose evaluations fail. reeval leaves an interrupted run in the `.reeval/runs` directory. A run that `budget.duration` stops is recorded with status `budget`, and a later `reeval compare` skips a run with that status when it picks a base.

The commit step runs when `reeval run` passes or fails. `if: ${{ !cancelled() }}` skips it when the job is canceled.

The job skips pull requests opened from forks because their default token has no push access.

</Step>
<Step title="Fail on a regression">

In the `.github/workflows/reeval.yml` file, add these two steps. `reeval compare` reads the history directory and picks the newest complete run on an ancestor of the pull request's commit. The steps append the diff to the job summary and fail the job on a regression, and they run unless the job is canceled.

```yaml title=".github/workflows/reeval.yml"
      - if: ${{ !cancelled() }}
        run: npx reeval compare --md >> "$GITHUB_STEP_SUMMARY"
      - if: ${{ !cancelled() }}
        run: npx reeval compare --fail-on-regression
```

The gate fails when an evaluation `regressed`, or when a `new` or `can't compare` evaluation fails, so the first run with no earlier record fails only on a failing evaluation. For more information about those words, see [Compare with past runs](/results/compare/past-runs#fail-on-a-regression).

</Step>
</Steps>

For a runnable project whose workflow runs the evaluations and gates on `reeval compare` without pushing a record, see [`with-github-pr-gate`](https://github.com/invertase/reeval/tree/main/examples/with-github-pr-gate). For more information about CI variables, see [Environment variables](/reference/environment-variables).