---
title: Offline evals
description: >-
Cut the sandbox off from the network, or allow only the hosts you list, and
see what the agent tried to reach.
---
> ## Documentation Index
> Fetch the complete documentation index at: https://docs.reeval.ai/llms.txt
> Use this file to discover all available pages before exploring further.
**Goal:** check that an agent can do a task without the internet, or with only your package registries, and see every host it tried.
An `egress` policy says where the sandbox may connect. Under a restricted policy, the Docker container is detached from every network and the attempt's egress gateway is its only way out (see [Egress policies](/concepts/egress)).
## The files
```ts title="evals/offline.eval.ts"
import { evaluate, expect } from '{{ pkg }}'
evaluate(
'writes a CSV parser offline',
{ egress: 'deny-all' },
async (c) => {
await c.send(
'Create csv.js exporting parse(text) that handles quoted fields, ' +
'and csv.test.js using node:test. No dependencies.',
)
await expect(c.exec('node --test')).toSucceed()
},
)
```
```ts title="evals/installs.eval.ts"
import { DEFAULT_EGRESS_ALLOW, defineEnvironment, evaluate, expect } from
'{{ pkg }}'
const registriesOnly = defineEnvironment('registries-only', {
egress: { allow: [...DEFAULT_EGRESS_ALLOW, 'api.acme.dev'] },
})
evaluate(
'adds zod validation',
{ environment: registriesOnly },
async (c) => {
await c.send('Install zod and add a validated parseUser(input) to user.js.')
await expect(c.exec('npm ls zod')).toSucceed()
},
)
```
| Policy | The sandbox can reach |
|---|---|
| `'proxy'` (default) | Any host over HTTP(S), through the gateway |
| `'allow-all'` | Anything, by any protocol, mostly unrecorded |
| `'deny-all'` | Only the automatic hosts below |
| `{ allow: [...] }` | The listed hosts, plus the automatic hosts |
`DEFAULT_EGRESS_ALLOW` is the package registries and GitHub: `registry.npmjs.org`, `registry.yarnpkg.com`, `github.com`, `*.github.com`, `*.githubusercontent.com`, `pypi.org` and `files.pythonhosted.org`. `*.example.com` matches subdomains, not `example.com` itself.
## Check the plan
```bash
npx reeval run --dry-run
```
```text
plan 2 evals · 2 cells · 2 attempts · ~6m 00s serial (~3m 00s at concurrency 4)
eval harnesses variants runs attempts notes
evals/installs.eval.ts › adds zod validation 1 — 1 1 egress: allow registry.npmjs.org, registry.yarnpkg.com, github.com, *.github.com, *.githubusercontent.com, pypi.org, files.pythonhosted.org, api.acme.dev + auto: api2.cursor.sh, *.cursor.sh
evals/offline.eval.ts › writes a CSV parser offline 1 — 1 1 egress: deny-all + auto: api2.cursor.sh, *.cursor.sh
```
Each restricted eval lists its policy and the automatic hosts it adds.
## What the agent can still reach
`'deny-all'` means "nothing but what the attempt itself needs", so the agent still works:
| Automatic host | Such as |
|---|---|
| The harness's API | `api2.cursor.sh` and `*.cursor.sh` for Cursor, `api.anthropic.com` and `*.anthropic.com` for Claude Code, `generativelanguage.googleapis.com` for Antigravity, Goose and pi, `api.openai.com` for Codex, `ai-gateway.vercel.sh` or `api.openai.com` for fx, `api.x.ai` for Grok Build, and the provider's hosts and `mcp.exa.ai` for OpenCode |
| Every brokered secret's `hosts` | A key the agent is meant to use keeps working |
| Each MCP `http` or `sse` server's host | The server the environment gives the agent |
The agent's own web tools may not go through the sandbox at all. Cursor's web search and fetch run in Cursor's backend, and Codex's on OpenAI's side, so an egress policy doesn't stop them and the egress record doesn't see them. fx's run in the sandbox but bypass the gateway, so a restricted policy blocks them unrecorded.
reeval still records them in the transcript, so check them there: `expect(turn).not.toHaveSearched()` fails an eval whose agent searched anyway. A search the gateway refuses is still a `web-search` (`ok: false`), so the matcher sees it too.
## Run it and see what was blocked
```bash
npx reeval run offline -v
```
A request to any other host gets a `403` from the gateway and the attempt carries on; your checks decide whether it passed. `-v` prints one `gateway` line per request. This one is from a real Antigravity run under an allowlist:
```text
… 14:00:24 gateway CONNECT antigravity-unleash.goog:443 → 403 · blocked: antigravity-unleash.goog isn't in the allowlist · ↑0 B ↓0 B · 2ms · no key substituted
```
That host is Antigravity's feature-flag service, which its CLI contacts in every session. reeval now lists hosts like it as expected blocks: still refused, but summarised in one `expected blocks` line at teardown instead of a line per request.
The record keeps every attempt's traffic under `egress`:
```bash
npx reeval show latest --json \
| jq -c '.evals[].attempts[] | {key, blocked: .egress.blocked}'
```
| Field | What it holds |
|---|---|
| `policy`, `allowed` | The policy, and every host a restricted policy let through |
| `contacted` | Each `host:port` reached, with connections and bytes sent and received |
| `blocked` | Each refused `host:port`, with connections and the reason |
| `expectedBlocked` | Refused hosts the harness's CLI always tries and nothing needs |
In the studio, the attempt's State tab shows the same panel:
## It needs Docker
The fake sandbox runs commands on the host, so it can't enforce a policy, and reeval refuses before anything starts:
```bash
npx reeval run --config reeval.fake.config.ts --dry-run
```
```text
reeval: evals/offline.eval.ts: evaluate("writes a CSV parser offline") runs under egress 'deny-all', which needs turning egress off, but the fake sandbox enforces no egress policy (its commands run on the host). Run it in docker(), or set egress: 'allow-all'.
```
An allowlist is checked by host name when the connection opens. An HTTPS tunnel to an allowed host that shares a CDN with others isn't inspected, so treat an allowlist as a guard against accidents, not against a determined agent.
## Variations
| Variation | How |
|---|---|
| One policy for the whole suite | Set `egress` in `reeval.config.ts` or a profile's `defaults`; groups, environments and evals can still override it |
| Offline as a compare | `compare: { baseline: online, variants: [offline] }` with `defineEnvironment('online', {})` and `defineEnvironment('offline', { egress: 'deny-all' })` shows how much the agent relies on the network, as lift |
| Your API only | `{ allow: ['api.acme.dev'] }` without `DEFAULT_EGRESS_ALLOW` blocks installs as well |
| A brokered key | `secret.fromEnv('ACME_TOKEN', { hosts: ['api.acme.dev'] })` adds `api.acme.dev` to the automatic hosts and keeps the token out of the sandbox |
## Next
What each policy enforces, and what's recorded.
Brokered keys and allowlists together.