> ## Documentation Index
> Fetch the complete documentation index at: https://mcpjam-docs-coming-from-mcp-inspector.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Running Evals

> Measure tool accuracy with statistical evaluations

LLMs are probabilistic—the same prompt might produce different results each time. Running a test once tells you it *can* work, not that it *reliably* works. Evals run the same test many times and measure statistical accuracy.

## Why Run Evals?

A single test pass can be misleading:

* The LLM might get lucky on one attempt
* Temperature introduces randomness
* Different phrasings might fail where others succeed

Running 30+ iterations gives you confidence in real-world performance:

* "This tool is called correctly 97% of the time"
* "Arguments are correct in 90% of cases"
* "Average latency is 1.2 seconds"

## EvalTest: Single Test Scenario

`EvalTest` runs one test function multiple times:

```typescript theme={null}
import { EvalTest } from "@mcpjam/sdk";

const test = new EvalTest({
  name: "addition-accuracy",
  test: async (agent) => {
    const result = await agent.run("Add 2 and 3");
    return result.hasToolCall("add"); // Return true/false
  },
});

const result = await test.run(agent, { iterations: 30 });

console.log(`Accuracy: ${(test.accuracy() * 100).toFixed(1)}%`);
// "Accuracy: 96.7%"
console.log(`${result.successes}/${result.iterations} passed`);
```

### Writing Test Functions

Test functions receive an `HostExecutor` (implemented by `HostRunner` and mock agents) and return `boolean`:

```typescript theme={null}
// Simple: did the right tool get called?
test: async (agent) => {
  const result = await agent.run("Add 5 and 3");
  return result.hasToolCall("add");
}

// Detailed: check arguments too
test: async (agent) => {
  const result = await agent.run("Add 10 and 20");
  const args = result.getToolArguments("add");
  return args?.a === 10 && args?.b === 20;
}

// Complex: multi-step workflow
test: async (agent) => {
  const r1 = await agent.run("Create a project called 'Test'");
  const r2 = await agent.run("Add a task to it", { context: r1 });
  return r1.hasToolCall("createProject") && r2.hasToolCall("createTask");
}
```

### Run Options

```typescript theme={null}
await test.run(agent, {
  iterations: 30,      // How many times to run
  concurrency: 5,      // Parallel runs (careful with rate limits)
  retries: 2,          // Retry failures
  timeoutMs: 30000,    // Per-test timeout
  mcpjam: {
    // Auto-save is enabled when MCPJAM_API_KEY is available
    suiteName: "SDK eval smoke",
    strict: false, // warn by default; true to fail CI on upload errors
  },
  onProgress: (done, total) => {
    console.log(`${done}/${total}`);
  },
});
```

### Metrics

After running, access various metrics:

```typescript theme={null}
test.accuracy();           // Success rate (0.0 - 1.0)
test.precision();          // Precision metric
test.recall();             // Recall metric
test.averageTokenUse();    // Avg tokens per iteration
```

## EvalSuite: Multiple Tests

Group related tests together:

```typescript theme={null}
import { EvalSuite, EvalTest } from "@mcpjam/sdk";

const suite = new EvalSuite({ name: "Math Operations" });

suite.add(new EvalTest({
  name: "addition",
  test: async (agent) => {
    const r = await agent.run("Add 5 and 3");
    return r.hasToolCall("add");
  },
}));

suite.add(new EvalTest({
  name: "multiplication",
  test: async (agent) => {
    const r = await agent.run("Multiply 4 by 6");
    return r.hasToolCall("multiply");
  },
}));

suite.add(new EvalTest({
  name: "division",
  test: async (agent) => {
    const r = await agent.run("Divide 20 by 4");
    return r.hasToolCall("divide");
  },
}));

const result = await suite.run(agent, { iterations: 30 });
console.log(`Overall: ${(result.aggregate.accuracy * 100).toFixed(1)}%`);
```

## Save Results to MCPJam

Both `EvalTest` and `EvalSuite` can automatically save results to MCPJam when a run completes. Set `MCPJAM_API_KEY` to an **MCPJam API key (`sk_…`)** from Settings → API keys and results are saved automatically:

```typescript theme={null}
await test.run(agent, {
  iterations: 30,
  mcpjam: {
    suiteName: "Addition Eval",
    passCriteria: { minimumPassRate: 90 },
  },
});
```

Results are filed under the project in `MCPJAM_PROJECT_ID` (or the `project` option), defaulting to your organization's **Default** project. For manual save APIs, CI metadata, and artifact uploads, see the [Save Results to MCPJam](/sdk/concepts/saving-results) guide.

### Tool errors vs. `passed` on upload

By default, when results are **built from traces** (auto-save and reporter helpers), a failed tool execution (MCP `isError`, errored tool spans, or error tool-results in messages) sets `passed: false` for that iteration even if your test function returned `true`. That keeps CI pass rate aligned with real server behavior.

To disable that for a run (for example you only assert which tools were called, not that every call succeeded), set `failOnToolError: false` on the `mcpjam` block:

```typescript theme={null}
await test.run(agent, {
  iterations: 30,
  mcpjam: {
    suiteName: "Addition Eval",
    passCriteria: { minimumPassRate: 90 },
    failOnToolError: false,
  },
});
```

See [Eval reporting — Tool execution and `passed`](/sdk/reference/eval-reporting#tool-execution-and-passed) for full detail and [`createEvalRunReporter`](/sdk/reference/eval-reporting#createevalrunreporter) overrides.

### Predicate gate

`successPredicates` lets you express richer pass/fail criteria beyond tool-call matching — without an LLM judge. Predicates are pure functions of the iteration transcript, so they produce the same verdict every time and are safe to use as a CI release gate.

A case passes iff **all** predicates pass. An absent or empty list means no predicate gate.

The eight built-in predicate types:

| Type                            | What it checks                                                            |
| ------------------------------- | ------------------------------------------------------------------------- |
| `toolCalledWith`                | A specific tool was called with matching args (partial/exact/ignore mode) |
| `toolCalledAtLeastOnce`         | A tool was called at least once                                           |
| `toolNeverCalled`               | A forbidden tool was never called                                         |
| `responseContains`              | The final assistant message contains a substring                          |
| `responseMatches`               | The final assistant message matches a regex                               |
| `noToolErrors`                  | No tool produced an error (MCP `isError` or transport failure)            |
| `finalAssistantMessageNonEmpty` | The final assistant message is non-empty                                  |
| `tokenBudgetUnder`              | Total token usage is strictly under a budget                              |

Predicates are authored in the corpus JSON (V1) and threaded through the runner automatically. See [Predicate gate reference](/sdk/reference/eval-reporting#predicate-gate) for the full type definitions and a code example.

### Suite Results

```typescript theme={null}
// Overall accuracy
console.log(`Suite: ${(suite.accuracy() * 100).toFixed(1)}%`);

// Per-test breakdown
for (const test of suite.getAll()) {
  console.log(`  ${test.getName()}: ${(test.accuracy() * 100).toFixed(1)}%`);
}

// Get specific test
const addTest = suite.get("addition");
console.log(`Addition accuracy: ${addTest.accuracy()}`);
```

## Choosing Iteration Count

| Scenario         | Iterations | Why                              |
| ---------------- | ---------- | -------------------------------- |
| Quick smoke test | 10         | Fast feedback during development |
| Regular testing  | 30         | Good statistical significance    |
| Pre-release      | 50-100     | High confidence before shipping  |
| Benchmarking     | 100+       | Comparing models or changes      |

## Best Practices

### Use Low Temperature

More deterministic results for testing:

```typescript theme={null}
const agent = new HostRunner({
  // ...
  temperature: 0.1,
});
```

### Handle Rate Limits

Reduce concurrency for rate-limited APIs:

```typescript theme={null}
await suite.run(agent, {
  iterations: 30,
  concurrency: 2, // Avoid hitting rate limits
});
```

### Test Edge Cases

Don't just test the happy path:

```typescript theme={null}
suite.add(new EvalTest({
  name: "handles-empty-input",
  test: async (agent) => {
    const r = await agent.run("Add numbers"); // No numbers given
    return !r.hasError(); // Should handle gracefully
  },
}));

suite.add(new EvalTest({
  name: "handles-large-numbers",
  test: async (agent) => {
    const r = await agent.run("Add 999999999 and 1");
    return r.hasToolCall("add");
  },
}));
```

### Set Quality Thresholds

Fail CI if accuracy drops below a threshold:

```typescript theme={null}
await suite.run(agent, { iterations: 30 });

if (suite.accuracy() < 0.90) {
  console.error(`Accuracy ${suite.accuracy()} below 90% threshold`);
  process.exit(1);
}
```

## Bring your own host

`HostRunner` is the right starting point when you just need an LLM + tools + a model string. For richer setups — pinning a host style/profile, applying sandbox/permission policy, or running the same configuration in CI that the MCPJam Inspector uses for its Playground — start from a `Host` spec instead:

```typescript theme={null}
import { MCPClientManager, Host, EvalTest } from "@mcpjam/sdk";

const manager = new MCPClientManager();
await manager.connectToServer("everything", {
  command: "npx",
  args: ["-y", "@modelcontextprotocol/server-everything"],
});

const host = new Host({
  style: "mcpjam",
  model: "openai/gpt-4o",
  systemPrompt: "You are a helpful assistant.",
}).requireServer("everything");

// Bind the spec to the live manager. apiKey lives on the runtime, not per-call.
const runtime = host.withManager(manager, {
  apiKey: process.env.OPENAI_API_KEY!,
});

const test = new EvalTest({
  name: "add",
  test: async (runner) => (await runner.run("Add 2+3")).hasToolCall("add"),
});
await test.run(runtime, { iterations: 10 });
```

`HostRuntime` (returned by `host.withManager(...)`) snapshots the live `Host` on every `.run()`, so mutating the bound host between iterations takes effect on the next iteration. `HostRunner` snapshots once at construction — pick `HostRunner` for a static spec, `HostRuntime` when you want the spec editable across calls.

**Personal computer.** A `Host` can declare a personal cloud workstation via the `computer` property (type `HostComputer`, exported from `@mcpjam/sdk`). When set, the chatbox surfaces a `bash` tool and a web terminal for signed-in members. Use `setComputer()` to attach or detach it fluently:

```typescript theme={null}
import { Host, type HostComputer } from "@mcpjam/sdk";

// Attach with the default shape
const host = new Host({ style: "mcpjam", model: "openai/gpt-4o" })
  .setComputer();

// Attach with an initial working directory
host.setComputer({ kind: "personal", toolset: "bash", workdir: "/srv/app" });

// Detach
host.setComputer(null);
```

Or pass it directly in the constructor:

```typescript theme={null}
const host = new Host({
  style: "mcpjam",
  model: "openai/gpt-4o",
  computer: { kind: "personal", toolset: "bash" },
});
```

`null` and an absent `computer` field hash identically, so existing host configs are unaffected. The `computer` field is automatically stripped from eval wire configs — evals never target personal computers.

**Statelessness.** `HostRuntime.run()` does NOT auto-replay history across turns. Use `PromptOptions.context` to thread context explicitly:

```typescript theme={null}
const r1 = await runtime.run("Who am I?");
const r2 = await runtime.run("List my projects", { context: [r1] });
```

**Eval reporting (Stage 5).** When you upload results to MCPJam, the reporter additionally sends the canonical host config alongside each run (`{ hostConfig, hostConfigHash }`) so the persisted row in the Evals UI shows the exact host you ran with. Pass 1 sends a run-level snapshot only when all iteration snapshots match; heterogeneous runs (mutating the bound `Host` between iterations) omit the run-level field for now. Old backends without the capability are unaffected. Source order is `iteration.hostSnapshot` → `executor.getHostSnapshot?.()` → `MCPJamReportingConfig.host`.

## Generate evals from the Inspector

You can also generate eval code from the MCPJam Inspector. Click **⋮ → Copy markdown for evals** on any server card, then paste it into an LLM. See the [Quickstart](/sdk/quickstart#generate-evals-from-the-inspector) for details.

## Next Steps

<CardGroup cols={2}>
  <Card title="Testing Across Providers" icon="cpu" href="/sdk/concepts/multi-provider">
    Compare performance across LLMs
  </Card>

  <Card title="EvalTest Reference" icon="book" href="/sdk/reference/eval-test">
    Full EvalTest API
  </Card>

  <Card title="EvalSuite Reference" icon="book" href="/sdk/reference/eval-suite">
    Full EvalSuite API
  </Card>

  <Card title="Save Results to MCPJam" icon="cloud-upload" href="/sdk/concepts/saving-results">
    Auto-save, manual APIs, CI metadata, and artifact upload
  </Card>
</CardGroup>