Running Evals

Measure tool accuracy with statistical evaluations

LLMs are probabilistic—the same prompt might produce different results each time. Running a test once tells you it can work, not that it reliably works. Evals run the same test many times and measure statistical accuracy.

Why Run Evals?

A single test pass can be misleading:

• The LLM might get lucky on one attempt
• Temperature introduces randomness
• Different phrasings might fail where others succeed

Running 30+ iterations gives you confidence in real-world performance:

• “This tool is called correctly 97% of the time”
• “Arguments are correct in 90% of cases”
• “Average latency is 1.2 seconds”

EvalTest: Single Test Scenario

EvalTest runs one test function multiple times:

import { EvalTest } from "@mcpjam/sdk";

const test = new EvalTest({
  name: "addition-accuracy",
  test: async (agent) => {
    const result = await agent.run("Add 2 and 3");
    return result.hasToolCall("add"); // Return true/false
  },
});

const result = await test.run(agent, { iterations: 30 });

console.log(`Accuracy: ${(test.accuracy() * 100).toFixed(1)}%`);
// "Accuracy: 96.7%"
console.log(`${result.successes}/${result.iterations} passed`);

Writing Test Functions

Test functions receive an HostExecutor (implemented by HostRunner and mock agents) and return boolean:

// Simple: did the right tool get called?
test: async (agent) => {
  const result = await agent.run("Add 5 and 3");
  return result.hasToolCall("add");
}

// Detailed: check arguments too
test: async (agent) => {
  const result = await agent.run("Add 10 and 20");
  const args = result.getToolArguments("add");
  return args?.a === 10 && args?.b === 20;
}

// Complex: multi-step workflow
test: async (agent) => {
  const r1 = await agent.run("Create a project called 'Test'");
  const r2 = await agent.run("Add a task to it", { context: r1 });
  return r1.hasToolCall("createProject") && r2.hasToolCall("createTask");
}

Run Options

await test.run(agent, {
  iterations: 30,      // How many times to run
  concurrency: 5,      // Parallel runs (careful with rate limits)
  retries: 2,          // Retry failures
  timeoutMs: 30000,    // Per-test timeout
  mcpjam: {
    // Auto-save is enabled when MCPJAM_API_KEY is available
    suiteName: "SDK eval smoke",
    strict: false, // warn by default; true to fail CI on upload errors
  },
  onProgress: (done, total) => {
    console.log(`${done}/${total}`);
  },
});

Metrics

After running, access various metrics:

test.accuracy();           // Success rate (0.0 - 1.0)
test.precision();          // Precision metric
test.recall();             // Recall metric
test.averageTokenUse();    // Avg tokens per iteration

EvalSuite: Multiple Tests

Group related tests together:

import { EvalSuite, EvalTest } from "@mcpjam/sdk";

const suite = new EvalSuite({ name: "Math Operations" });

suite.add(new EvalTest({
  name: "addition",
  test: async (agent) => {
    const r = await agent.run("Add 5 and 3");
    return r.hasToolCall("add");
  },
}));

suite.add(new EvalTest({
  name: "multiplication",
  test: async (agent) => {
    const r = await agent.run("Multiply 4 by 6");
    return r.hasToolCall("multiply");
  },
}));

suite.add(new EvalTest({
  name: "division",
  test: async (agent) => {
    const r = await agent.run("Divide 20 by 4");
    return r.hasToolCall("divide");
  },
}));

const result = await suite.run(agent, { iterations: 30 });
console.log(`Overall: ${(result.aggregate.accuracy * 100).toFixed(1)}%`);

Save Results to MCPJam

Both EvalTest and EvalSuite can automatically save results to MCPJam when a run completes. Set MCPJAM_API_KEY to an MCPJam API key (sk_…) from Settings → API keys and results are saved automatically:

await test.run(agent, {
  iterations: 30,
  mcpjam: {
    suiteName: "Addition Eval",
    passCriteria: { minimumPassRate: 90 },
  },
});

Results are filed under the project in MCPJAM_PROJECT_ID (or the project option), defaulting to your organization’s Default project. For manual save APIs, CI metadata, and artifact uploads, see the Save Results to MCPJam guide.

Tool errors vs. passed on upload

By default, when results are built from traces (auto-save and reporter helpers), a failed tool execution (MCP isError, errored tool spans, or error tool-results in messages) sets passed: false for that iteration even if your test function returned true. That keeps CI pass rate aligned with real server behavior.

To disable that for a run (for example you only assert which tools were called, not that every call succeeded), set failOnToolError: false on the mcpjam block:

await test.run(agent, {
  iterations: 30,
  mcpjam: {
    suiteName: "Addition Eval",
    passCriteria: { minimumPassRate: 90 },
    failOnToolError: false,
  },
});

See Eval reporting — Tool execution and passed for full detail and createEvalRunReporter overrides.

Predicate gate

successPredicates lets you express richer pass/fail criteria beyond tool-call matching — without an LLM judge. Predicates are pure functions of the iteration transcript, so they produce the same verdict every time and are safe to use as a CI release gate.

A case passes iff all predicates pass. An absent or empty list means no predicate gate.

The eight built-in predicate types:

Type	What it checks
toolCalledWith	A specific tool was called with matching args (partial/exact/ignore mode)
toolCalledAtLeastOnce	A tool was called at least once
toolNeverCalled	A forbidden tool was never called
responseContains	The final assistant message contains a substring
responseMatches	The final assistant message matches a regex
noToolErrors	No tool produced an error (MCP isError or transport failure)
finalAssistantMessageNonEmpty	The final assistant message is non-empty
tokenBudgetUnder	Total token usage is strictly under a budget

Predicates are authored in the corpus JSON (V1) and threaded through the runner automatically. See Predicate gate reference for the full type definitions and a code example.

Suite Results

// Overall accuracy
console.log(`Suite: ${(suite.accuracy() * 100).toFixed(1)}%`);

// Per-test breakdown
for (const test of suite.getAll()) {
  console.log(`  ${test.getName()}: ${(test.accuracy() * 100).toFixed(1)}%`);
}

// Get specific test
const addTest = suite.get("addition");
console.log(`Addition accuracy: ${addTest.accuracy()}`);

Choosing Iteration Count

Scenario	Iterations	Why
Quick smoke test	10	Fast feedback during development
Regular testing	30	Good statistical significance
Pre-release	50-100	High confidence before shipping
Benchmarking	100+	Comparing models or changes

Best Practices

Use Low Temperature

More deterministic results for testing:

const agent = new HostRunner({
  // ...
  temperature: 0.1,
});

Handle Rate Limits

Reduce concurrency for rate-limited APIs:

await suite.run(agent, {
  iterations: 30,
  concurrency: 2, // Avoid hitting rate limits
});

Test Edge Cases

Don’t just test the happy path:

suite.add(new EvalTest({
  name: "handles-empty-input",
  test: async (agent) => {
    const r = await agent.run("Add numbers"); // No numbers given
    return !r.hasError(); // Should handle gracefully
  },
}));

suite.add(new EvalTest({
  name: "handles-large-numbers",
  test: async (agent) => {
    const r = await agent.run("Add 999999999 and 1");
    return r.hasToolCall("add");
  },
}));

Set Quality Thresholds

Fail CI if accuracy drops below a threshold:

await suite.run(agent, { iterations: 30 });

if (suite.accuracy() < 0.90) {
  console.error(`Accuracy ${suite.accuracy()} below 90% threshold`);
  process.exit(1);
}

Bring your own host

HostRunner is the right starting point when you just need an LLM + tools + a model string. For richer setups — pinning a host style/profile, applying sandbox/permission policy, or running the same configuration in CI that the MCPJam Inspector uses for its Playground — start from a Host spec instead:

import { MCPClientManager, Host, EvalTest } from "@mcpjam/sdk";

const manager = new MCPClientManager();
await manager.connectToServer("everything", {
  command: "npx",
  args: ["-y", "@modelcontextprotocol/server-everything"],
});

const host = new Host({
  style: "mcpjam",
  model: "openai/gpt-4o",
  systemPrompt: "You are a helpful assistant.",
}).requireServer("everything");

// Bind the spec to the live manager. apiKey lives on the runtime, not per-call.
const runtime = host.withManager(manager, {
  apiKey: process.env.OPENAI_API_KEY!,
});

const test = new EvalTest({
  name: "add",
  test: async (runner) => (await runner.run("Add 2+3")).hasToolCall("add"),
});
await test.run(runtime, { iterations: 10 });

HostRuntime (returned by host.withManager(...)) snapshots the live Host on every .run(), so mutating the bound host between iterations takes effect on the next iteration. HostRunner snapshots once at construction — pick HostRunner for a static spec, HostRuntime when you want the spec editable across calls.

Personal computer. A Host can declare a personal cloud workstation via the computer property (type HostComputer, exported from @mcpjam/sdk). When set, the chatbox surfaces a bash tool and a web terminal for signed-in members. Use setComputer() to attach or detach it fluently:

import { Host, type HostComputer } from "@mcpjam/sdk";

// Attach with the default shape
const host = new Host({ style: "mcpjam", model: "openai/gpt-4o" })
  .setComputer();

// Attach with an initial working directory
host.setComputer({ kind: "personal", toolset: "bash", workdir: "/srv/app" });

// Detach
host.setComputer(null);

Or pass it directly in the constructor:

const host = new Host({
  style: "mcpjam",
  model: "openai/gpt-4o",
  computer: { kind: "personal", toolset: "bash" },
});

null and an absent computer field hash identically, so existing host configs are unaffected. The computer field is automatically stripped from eval wire configs — evals never target personal computers.

Statelessness. HostRuntime.run() does NOT auto-replay history across turns. Use PromptOptions.context to thread context explicitly:

const r1 = await runtime.run("Who am I?");
const r2 = await runtime.run("List my projects", { context: [r1] });

Eval reporting (Stage 5). When you upload results to MCPJam, the reporter additionally sends the canonical host config alongside each run ({ hostConfig, hostConfigHash }) so the persisted row in the Evals UI shows the exact host you ran with. Pass 1 sends a run-level snapshot only when all iteration snapshots match; heterogeneous runs (mutating the bound Host between iterations) omit the run-level field for now. Old backends without the capability are unaffected. Source order is iteration.hostSnapshot → executor.getHostSnapshot?.() → MCPJamReportingConfig.host.

Generate evals from the Inspector

You can also generate eval code from the MCPJam Inspector. Click ⋮ → Copy markdown for evals on any server card, then paste it into an LLM. See the Quickstart for details.

Next Steps

Testing Across Providers

Compare performance across LLMs

EvalTest Reference

Full EvalTest API

EvalSuite Reference

Full EvalSuite API

Save Results to MCPJam

Auto-save, manual APIs, CI metadata, and artifact upload