--- title: Sandboxes and toolchains description: >- Run the agent on Python, your own Dockerfile or reeval's image with extra tools, using docker(). --- > ## Documentation Index > Fetch the complete documentation index at: https://docs.reeval.ai/llms.txt > Use this file to discover all available pages before exploring further. Every attempt runs in a fresh Docker container. `docker()` with no options uses reeval's image: Node 22, git and the harness's agent. When the task needs another toolchain, give the sandbox another base. ```ts import { docker } from '{{ pkg }}/sandbox' docker() // reeval's image docker({ from: 'python:3.12-slim' }) // reeval adds Node, git, the agent docker({ install: ['npm i -g firebase-tools@14'] }) // extra tools docker({ dockerfile: './sandbox/Dockerfile' }) // your own, from this file docker({ image: 'ghcr.io/acme/agent:1' }) // already has the agent ``` A sandbox has one base (`image`, `from` or `dockerfile`), and optionally `install` commands on top. What's baked into the image belongs to the sandbox. What each attempt adds, such as files, secrets and setup steps, belongs to the [environment](/guide/environments). ## Start from a toolchain image ```ts title="evals/python.eval.ts" import { defineEnvironment, evaluate, expect } from '{{ pkg }}' import { docker } from '{{ pkg }}/sandbox' const python = defineEnvironment('python', { sandbox: docker({ from: 'python:3.12-slim', id: 'python' }), }) evaluate( 'writes and tests fizzbuzz in python', { environment: python }, async (c) => { await c.send( 'Write fizzbuzz.py with a function fizzbuzz(n) returning the usual ' + 'FizzBuzz string for n, and test_fizzbuzz.py with unittest tests ' + 'for 3, 5, 15 and 7. Standard library only.', ) await expect(c.exec('python3 --version')).toSucceed() await expect(c.exec('python3 -m unittest -q')).toSucceed() }, ) ``` `from` takes any Debian or Ubuntu based image. reeval bakes Node, pnpm, git and the harness's agent on top the first time, and caches the result. `id: 'python'` names the sandbox in the cell key. ## Add tools with install ```ts title="evals/jq.eval.ts" const jq = defineEnvironment('jq', { sandbox: docker({ install: [ 'apt-get update && apt-get install -y --no-install-recommends jq ' + '&& rm -rf /var/lib/apt/lists/*', ], }), }) evaluate('summarises a JSON file with jq', { environment: jq }, async (c) => { // … await expect(c.exec('jq --version')).toSucceed() await expect(c.exec('./count.sh')).toSucceed() }) ``` `install` runs its commands in order, as root, in one build container, with `HOME` set to `/home/reeval`: what an installer puts in `~` is where the agent runs, and reeval hands it to the `reeval` user. reeval caches the result as one image, keyed on the base image and the exact list, so any edit builds anew. Install commands never see secrets or the workspace; anything that needs them is an environment `setup` step. ## Bring a Dockerfile ```dockerfile title="docker/sqlite/Dockerfile" ARG REEVAL_BASE FROM ${REEVAL_BASE} RUN apt-get update \ && apt-get install -y --no-install-recommends sqlite3 \ && rm -rf /var/lib/apt/lists/* COPY seed.sql /opt/seed.sql ``` ```ts title="evals/sqlite.eval.ts" const sqlite = defineEnvironment('sqlite', { sandbox: docker({ dockerfile: '../docker/sqlite/Dockerfile' }), }) evaluate( 'queries a seeded SQLite database', { environment: sqlite }, async (c) => { // … const report = await c.exec('sqlite3 app.db < report.sql') await expect(report).toSucceed() expect(report.stdout).toBe('Ada\nGrace\nLinus\n') }, ) ``` reeval builds the Dockerfile from its directory with `docker build`, and passes `REEVAL_BASE`: its own image for the harness. `FROM ${REEVAL_BASE}` layers on it, so the result already has the harness and is used as is. From any other Debian or Ubuntu base, reeval adds the harness on top, as it does for `from`. Keep `root` as the user and leave `/reeval` alone in your Dockerfile. `RUN` lines build as root, but the agent runs as the `reeval` user (see [Sandbox security](#sandbox-security)), so install tools system-wide or under `/home/reeval` and `chown` them to `reeval`. `install` doesn't combine with `dockerfile`: write the commands as `RUN` lines. ## Build the images ```bash npx reeval image build --dry-run ``` ```text images 7 sandbox images (nothing built: --dry-run) docker · cursor · 23 evals + base reeval-sandbox:cursor-0.1.6 (to build: not built yet) docker · claude-code · 1 eval + base reeval-sandbox:claude-code-1.0.160-patch.b17cde50 (to build: not built yet) … docker+0ab0e263 · cursor + install · 1 eval + base reeval-sandbox:cursor-0.1.6 (to build: not built yet) + install 1 command reeval-install: (to build: not built yet) python · cursor · 1 eval + from python:3.12-slim reeval-sandbox:cursor-0.1.6-python-3-12-slim-375f4a61 (to build: on python:3.12-slim (a6e34c598f24)) … docker:sqlite · cursor · 1 eval + dockerfile docker/sqlite/Dockerfile reeval-dockerfile:sqlite- (to build: not built yet (reeval-dockerfile:sqlite-9efec1fbf61588d3)) + harness reeval-dockerfile:sqlite- (to build: reeval-sandbox:cursor-0.1.6 not built yet) ``` Each step says how it stands: `✓` ready, `+` to build, `~` stale, `✗` missing. `reeval image build` builds what's missing or stale, three at a time, and reports every image that failed (exit 3). Filters, `--profile` and `REEVAL_PROFILE` apply; `--tag` doesn't. You don't have to run it first: `reeval run` builds whatever is missing before any attempt starts, and `reeval run --dry-run` ends with the same check, one line per image. A `from` tag such as `node:22` is looked up once, and reeval builds on the digest it names then. Later runs reuse that build even when the tag moves; `--dry-run` says when it has, and `reeval image build` rebuilds it. Pin a digest (`node:22@sha256:…`) to never move, or run `reeval image build --force` to rebuild and pull everything. ## When an image fails ```bash npx reeval run --ui plain ``` ```text plan run 20261020T214011Z_… · 2 evals · 2 cells · 2 attempts · ~2s serial (~1s at concurrency 4) preparing 2 sandboxes: docker, docker+048fc5d7 + install ✗ docker+048fc5d7 + install failed to prepare (set by evaluate("summarises a JSON file with jq"); only its attempts fail) Building the install layer (1 command) failed: install 1/1 exited 100: apt-get update && apt-get install -y jqq | … | Reading package lists... | Building dependency tree... | Reading state information... | E: Unable to locate package jqq (full log: .reeval/cache/images/reeval-install-e19a593383099e07.log) … pass greets infra error summarises a JSON file with jq infra error Building the install layer (1 command) failed: install 1/1 exited 100: apt-get update && apt-get install -y jqq 5s · results .reeval/runs/20261020T214011Z_…/summary.md 1 pass · 1 infra error Set analysis.model in reeval.config.ts for answers to what each eval asks. ``` An image that fails fails only the attempts that need it, once each: they're infra errors that name the image and the level that set it, and everything else runs. When the config (or a profile) sets the failing sandbox, every eval needs it, so `reeval run` stops before any attempt with exit 3 (exit 2 for a missing `docker({ image })`). The headline says which step failed and why, then the step's own last lines and the build log, and names out of disk space and exit 137 (out of memory). Network and registry errors are retried twice; a failing `install` or `RUN` step isn't, since it would fail the same way. The attempt's record has the failure in `error.prepare`. ## Run them ```bash npx reeval run python sqlite jq --log-order grouped --output-logs full ``` ```text plan run 20261020T214032Z_… · 3 evals · 3 cells · 3 attempts · ~34s serial (~11s at concurrency 4) preparing 3 sandboxes: docker+0ab0e263 + install, python, docker:sqlite … [writes#1 cursor] ▶ evals/python.eval.ts › writes and tests fizzbuzz in python [writes#1 cursor] +0:00 sandbox ready 2s [writes#1 cursor] +0:02 turn 1 done 6s · 1 tool call, 2 file edits [writes#1 cursor] +0:08 $ python3 --version exit 0 65ms [writes#1 cursor] +0:08 $ python3 -m unittest -q exit 0 231ms … pass summarises a JSON file with jq pass writes and tests fizzbuzz in python pass queries a seeded SQLite database 37s · 167k in (79k cached) · 1,653 out · results .reeval/runs/20261020T214032Z_…/summary.md 3 pass ``` ## Sandbox security ```ts docker() // locked down by default docker({ memory: '8g', cpus: 4, pids: 4096 }) // raise the limits docker({ root: true }) // root at run time docker({ capabilities: ['NET_RAW'] }) // one extra capability ``` The agent runs code you didn't write, so each container is locked down without any options: | What | How it's isolated | |---|---| | Your keys | The harness's API key and brokered secrets never enter the container. The agent sees a placeholder, and the egress gateway on your machine swaps in the real value on the way out, only to that secret's hosts. See [credential brokering](/concepts/credential-brokering) | | The network | Under a restricted [egress policy](/guide/secrets-and-egress) the container is detached from every Docker network, so the gateway is its only way out. The agent can't edit routes or firewall rules, and the gateway's process runs as root, so the agent can't stop or replace it | | Privileges | The agent, setup steps and `c.exec` checks run as `reeval` (uid 10001, home `/home/reeval`), not root. That user holds no Linux capabilities and `no-new-privileges` is set. Image builds (`install`, your Dockerfile, `from`) still run as root | | Resources | A runaway agent or fork bomb hits the container's limits, not your machine (below) | | Limit | Default | Option | |---|---|---| | Memory | 4 GB. A process past it and its swap is killed (exit 137) and the attempt is an infra error; the container keeps running | `memory: '8g'` | | Swap on top of memory | As much as `memory` (4 GB), if the host has swap to give. `0` makes `memory` the hard ceiling | `swap: '2g'` | | CPUs | 2, or fewer if the Docker host has fewer | `cpus: 4` | | Processes and threads | 1024 | `pids: 4096` | Everything else works as the `reeval` user: `npm i`, `pip install` (as a user install), `git`, `curl`, and `npm i -g`, which goes to `~/.local`, on `PATH`. ## What isn't isolated | Gap | What it means | |---|---| | It's a container, not a VM | It shares your machine's kernel. The steps above make a kernel or Docker bug much harder to reach, but they aren't a hardware boundary | | `allow-all` reaches anything | Only when an eval or the config sets it; the default, `proxy`, sends everything through the gateway. Under it the keys are still brokered, but the agent can go around the proxy to any host, cloud metadata and your LAN, unrecorded | | WebSocket messages aren't scrubbed | Headers are, but a brokered host that echoes a real secret back inside a WebSocket message after the upgrade shows it to the agent | | Secrets sent `to: 'checks'` are real values in setup steps and checks | Those run as the same `reeval` user as the agent, so a process the agent left running could read them while they run | ## When an eval needs more ```text WARNING: running as root in the sandbox: evaluate("installs deps") (docker({ root: true })) warning: extra capabilities in the sandbox: reeval.config.ts (docker({ capabilities: ['NET_RAW'] })) ``` `docker({ root: true })` runs the agent, setup and checks as root, with only the capabilities `apt-get` needs (`CHOWN`, `DAC_OVERRIDE`, `FOWNER`, `SETUID`, `SETGID`). Prefer baking packages in with `install`, which needs no opt-in. `capabilities` adds a specific one, such as `NET_RAW` for `ping`. Either one is printed at the start of every run that uses it, naming where it was set, and kept in the run record. ## Run on Podman ```ts title="evals/env.ts" import { defineEnvironment } from '{{ pkg }}' import { podman } from '{{ pkg }}/sandbox' export const python = defineEnvironment('python', { sandbox: podman({ from: 'python:3.12-slim' }), }) ``` `podman()` takes the same options as `docker()` and runs the same sandbox on Podman 4 or later. Every container joins Podman's bridge network with Docker's default seccomp profile, so rootless Podman confines it as Docker does, and a restricted egress policy fails closed if it can't. Podman keeps its own images; `reeval image build` and `reeval doctor` cover them. See [podman()](/reference/sandbox/podman). reeval is tested on rootless Podman on Linux, and on macOS with Podman 6.1 in a `podman machine` VM (`applehv`); Windows hasn't been tested yet. Give the machine 4 GB with `podman machine init --memory 4096` or `podman machine set --memory 4096`. See [macOS and Windows](/reference/sandbox/podman#macos-and-windows). ## One sandbox per eval | Cell key | Harness | Environment | Sandbox | |---|---|---|---| | `cursor:composer-2.5/-/python` | `cursor:composer-2.5` | none (`-`) | `python` | An eval has one sandbox, and every attempt gets a fresh container from its image. Comparing toolchains means different checks (`python3 -m unittest` against `node --test`), so write a separate eval for each. The sandbox is still part of each cell's key. ## Next Keep keys out of the sandbox and control where it connects. Complete toolchain setups to copy.