---
title: Nightly matrix and Pages
description: >-
Run every eval on several agents each night, commit the history to the repo
and publish the studio to GitHub Pages.
---
> ## Documentation Index
> Fetch the complete documentation index at: https://docs.reeval.ai/llms.txt
> Use this file to discover all available pages before exploring further.
**Goal:** a scheduled run of the whole suite, three attempts per cell on two agents, recorded into `evals/.history` on your main branch and published as a static studio anyone on the team can open.
The committed history is also what gives the [PR gate](/recipes/github-pr-gate) a base to compare with.
This workflow passes [actionlint](https://github.com/rhysd/actionlint), and each `reeval` command in it was run locally, but the workflow as a whole hasn't run in GitHub Actions. The action versions were the latest releases when this page was written.
## The files
The `nightly` profile adds a second harness, three runs per cell and a pass-rate gate. `record: 'always'` records the run even outside CI. The `pr` profile stays cheap.
```ts title="evals/harnesses.ts"
import { claudeCode, cursor } from '{{ pkg }}/harness'
export const composer = cursor({ model: 'composer-2.5' })
export const sonnet = claudeCode({ model: 'claude-sonnet-5' })
```
```ts title="reeval.config.ts"
import { defineConfig } from '{{ pkg }}'
import { docker } from '{{ pkg }}/sandbox'
import { composer, sonnet } from './evals/harnesses.ts'
export default defineConfig({
harness: composer,
judge: { model: 'google/gemini-3.8-flash', threshold: 0.8 },
defaultProfile: { ci: 'pr' },
profiles: {
pr: {
limit: { runs: 1, compare: false },
budget: { attempts: 20 },
},
nightly: {
defaults: {
runs: 3,
harness: [composer, sonnet],
gate: { passRate: 0.6 },
},
record: 'always',
budget: { attempts: 80, duration: '2h' },
},
},
})
```
The workflow has two jobs. `evals` runs eval code next to your API keys, so it gets no write token. `publish` gets the write token and runs no eval code.
```yaml title=".github/workflows/nightly.yml"
name: nightly
on:
schedule:
- cron: '17 3 * * *'
workflow_dispatch:
permissions:
contents: read
concurrency:
group: nightly
cancel-in-progress: false
jobs:
evals:
runs-on: ubuntu-24.04
# The nightly budget stops new attempts at 2h; leave room to finish.
timeout-minutes: 150
steps:
# Eval code runs on the host with the keys, so this job has no token.
- uses: actions/checkout@v7
with:
fetch-depth: 0
persist-credentials: false
- uses: actions/setup-node@v7
with:
node-version: 22
cache: npm
- run: npm ci
- name: Restore sandbox images
id: images
uses: actions/cache@v6
with:
path: ${{ runner.temp }}/reeval-images.tar
key: >-
reeval-images-nightly-${{ hashFiles('package-lock.json',
'reeval.config.ts', 'docker/**') }}
- name: Load sandbox images
if: steps.images.outputs.cache-hit == 'true'
run: docker load -i "$RUNNER_TEMP/reeval-images.tar"
- name: Build sandbox images
if: steps.images.outputs.cache-hit != 'true'
run: |
npx reeval image build --profile nightly
images=$(docker images --format '{{.Repository}}:{{.Tag}}' \
| grep '^reeval-')
docker save -o "$RUNNER_TEMP/reeval-images.tar" $images
- name: Run evals
env:
CI: 1
CURSOR_API_KEY: ${{ secrets.CURSOR_API_KEY }}
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
GOOGLE_GENERATIVE_AI_API_KEY: ${{ secrets.GOOGLE_AI_KEY }}
run: npx reeval run --profile nightly
- name: Upload the history
if: ${{ !cancelled() }}
uses: actions/upload-artifact@v7
with:
name: reeval-history
path: evals/.history
include-hidden-files: true
if-no-files-found: ignore
retention-days: 7
publish:
needs: evals
if: ${{ !cancelled() }}
runs-on: ubuntu-24.04
timeout-minutes: 15
permissions:
contents: write
pages: write
id-token: write
environment:
name: github-pages
url: ${{ steps.deploy.outputs.page_url }}
steps:
- uses: actions/checkout@v7
- uses: actions/setup-node@v7
with:
node-version: 22
cache: npm
- run: npm ci
- uses: actions/download-artifact@v8
with:
name: reeval-history
path: evals/.history
- name: Commit the history
run: |
git config user.name 'github-actions[bot]'
git config user.email \
'41898282+github-actions[bot]@users.noreply.github.com'
git add evals/.history
if git diff --cached --quiet; then exit 0; fi
git commit -m 'evals: record the nightly run'
git push
- run: npx reeval studio build --out studio-dist
- uses: actions/upload-pages-artifact@v5
with:
path: studio-dist
- id: deploy
uses: actions/deploy-pages@v5
```
Before the first run, set **Settings › Pages › Source** to **GitHub Actions**, and add the three secrets.
## Check the plan
A dry run of this config under the nightly profile, and under CI's default:
```bash
npx reeval run --profile nightly --dry-run
```
```text
plan profile nightly (--profile) · 2 evals · 4 cells · 12 attempts · ~36m 00s serial (~9m 00s at concurrency 4) · budget 80 attempts ✓, 2h
eval harnesses variants runs attempts notes
evals/hello.eval.ts › greet module › creates a tested greet module 2 — 3 6
evals/hello.eval.ts › greet module › adds a default name in a… 2 — 3 6
```
```bash
CI=1 npx reeval run --dry-run
```
```text
plan profile pr (auto: CI is set) · 2 evals · 2 cells · 2 attempts · ~6m 00s serial (~3m 00s at concurrency 4) · budget 20 attempts ✓
…
```
`reeval image build --profile nightly` builds an image per harness, Cursor's and Claude Code's. With `--dry-run` it lists them:
```text
images 2 sandbox images (nothing built: --dry-run)
docker · cursor · 2 evals
base reeval-sandbox:cursor-0.1.0 (rebuild: the harness bootstrap)
docker · claude-code · 2 evals
base reeval-sandbox:claude-code-1.0.160-patch.b17cde50 (rebuild: the harness bootstrap)
```
## Record and commit
At the end of a complete run, reeval copies it into the history folder and says so. This line is from a real Cursor run in CI:
```text
recorded evals/.history/20260926T174109Z_b010670_ci_8f60.json + 9 artifact files in evals/.history/20260926T174109Z_b010670_ci_8f60/ (30.2 KB). Commit it with git.
```
A run that fails evals is still complete, so it's recorded, and `publish` still runs because of `!cancelled()`. A run that hits `budget.duration` ends with status `budget` and is recorded too, with its unstarted attempts skipped, so the publish job commits what the budget allowed. `reeval compare` never takes it as a base.
If your main branch is protected, let the `github-actions[bot]` push to it, or change the commit step to open a pull request.
## Build the studio
```bash
npx reeval studio build --out studio-dist
```
```text
built ../capsite/: 2 runs, 83 files, 913 KB (records, transcripts, diffs, logs and judge inputs)
Serve it from any static host. Anyone who can open it can read every included file.
```
That output is from a project of greeting evals on the fake harness, with `--out /tmp/capsite`. The site uses hash routing and relative paths, so it works under a Pages project path.
The build includes transcripts, diffs and judge inputs. On a public Pages site, pass `--no-artifacts` to publish records and the index only.
## Variations
| Variation | How |
|---|---|
| Upload instead of commit | Drop the commit step and keep the `reeval-history` artifact; the studio still builds, but the PR gate has no committed base |
| Only the smoke evals | A `smoke` profile with `tags: ['smoke']` and a small `budget`, run first, catches a broken key before the long run |
| More agents | Add `antigravity({ model })` to the nightly `harness` list and `GEMINI_API_KEY` to the secrets; see [Compare agents](/recipes/compare-agents) |
| Another host | `studio-dist` is a plain static folder, so Netlify or Cloudflare Pages can serve it too |
## Next
What each page of the studio shows.
What a real suite found, and what it missed.