---
title: Test an SDK
description: See whether an agent can use an SDK correctly.
---
> ## Documentation Index
> Fetch the complete documentation index at: https://docs.reeval.ai/llms.txt
> Use this file to discover all available pages before exploring further.

An SDK test seeds the SDK into the workspace, asks the harness to build with it, and checks the result with a script the harness never sees. The sample tests the small Acme weather SDK. For a runnable project, see [`with-sdk-usage`](https://github.com/invertase/reeval/tree/main/examples/with-sdk-usage).

## Test the SDK in a package

<Steps>
<Step title="Write the evaluation">

Create the `evals/sdk.eval.ts` file with the environment, the task, the check script and a judge.

```ts title="evals/sdk.eval.ts"
import { readFileSync } from 'node:fs'
import { defineEnvironment, evaluate, expect } from '{{ pkg }}'

// An empty package with the Acme SDK seeded into vendor/acme.
const app = defineEnvironment('acme-app', ({ directory }) => ({
  files: {
    ...directory('./project'),
    ...directory('./acme-sdk', { into: 'vendor/acme' }),
  },
}))
const checkClient = readFileSync(
  new URL('./hidden/check-client.mjs', import.meta.url), 'utf8',
)

evaluate('uses the Acme SDK correctly', { environment: app }, async (c) => {
  await c.send(
    'Write src/forecast.js exporting forecast(city), which resolves to ' +
      'the temperature in Celsius from the Acme SDK in vendor/acme (see ' +
      'its README.md), with the key in ACME_API_KEY. The API sometimes ' +
      'answers 503; retry those.',
  )

  await c.writeFile('check-client.mjs', checkClient)
  await expect(c.exec('node check-client.mjs')).toSucceed()
  expect(await c.judge(
    'Does forecast() use client.retry rather than its own loop?',
  )).toPass()
})
```

`directory('./acme-sdk', { into: 'vendor/acme' })` copies the SDK and its `README.md` into the workspace. For more information about seeding files, see [Place files](/configuration/environments/files).

`c.writeFile` adds `check-client.mjs` after the harness finishes, so the harness never sees it. The script runs `forecast()` against a stand-in API that answers `503` once. The judge asks the one question a command can't answer. For more information about judgments, see [Judge agent work](/evaluations/grading/judgments).

</Step>
<Step title="Run the evaluation">

From the directory that contains the `reeval.config.ts` file, run the `evals/sdk.eval.ts` file. The `sdk` filter matches the file path.

```bash
npx reeval run sdk
```

The run needs an API key for the harness and for the judge. For more information about each harness's key, see [Choose an agent](/configuration/agents). For more information about the judge's key, see [Judge agent work](/evaluations/grading/judgments).

</Step>
<Step title="Read the outcome">

When both assertions pass, the block for `uses the Acme SDK correctly` is one line. A `forecast()` that calls the SDK without retrying fails the check script.

```text
fail             uses the Acme SDK correctly
  toSucceed node check-client.mjs  failed
    expected command to succeed, exit code 1
```

The attempt's `checks/1.log`, under `.reeval/runs/<run>/`, has the command's output.

```text
$ node check-client.mjs
exit 1
…
AcmeError: Acme API answered 503
    at Object.weather (file:///…/work/vendor/acme/index.js:15:24)
    at async forecast (file:///…/work/src/forecast.js:5:11)
```

For more information about outcome words, see [Read the outcomes](/results/output#read-the-outcomes).

</Step>
</Steps>

## Next

For more information about file, command and transcript assertions, see [Assertions](/evaluations/grading/assertions).