> ## Documentation Index > Fetch the complete documentation index at: https://mcpjam-docs-coming-from-mcp-inspector.mintlify.site/llms.txt > Use this file to discover all available pages before exploring further. # Create an eval run (async) > Creates a suite run from an existing `suiteId` (rerun) and/or inline `tests`, then **detaches execution and responds `202` immediately** with the `runId`. Validation and quota errors surface on this request; poll `GET /eval-runs/{runId}` for progress. The run appears live in the hosted UI Runs tab, tagged `source: "api"`. A bare `suiteId` with no inline tests reruns the suite as configured. Per-organization concurrency is capped (default 2 concurrent runs); exceeding it returns `429` with `details.reason: "CONCURRENT_RUN_LIMIT"`. ## OpenAPI ````yaml /reference/openapi.json post /projects/{projectId}/eval-runs openapi: 3.1.0 info: title: MCPJam API version: 1.0.0-preview description: >- Programmatic access to MCP servers saved in your MCPJam projects — live diagnostics (validate, inspect, export) and operations: call tools, render prompts, run eval suites asynchronously and poll their results, and import OAuth tokens. **The API is in preview**: the surface may change while we finish the design. Error `code` values are stable; error `message` strings are not. Write clients that ignore unknown response fields. contact: name: MCPJam url: https://github.com/MCPJam/inspector/issues servers: - url: https://app.mcpjam.com/api/v1 description: Hosted MCPJam security: - bearerAuth: [] tags: - name: Hosts description: >- Project hosts: named model + capability profiles you run chats and eval suites against. - name: Computer environments description: >- Custom Computer images: digest-pinned Dockerfile environments your project's computers boot from. - name: Server diagnostics description: Connect-level health checks against a saved MCP server. - name: Primitives description: 'The server''s MCP primitives: tools, prompts, and resources.' - name: Export description: Full-server snapshots for diffing and CI. - name: Execution description: 'Run the server''s primitives: call tools, render prompts.' - name: Eval runs description: >- Asynchronous eval suite runs: create with 202, poll status, iterations, and traces. - name: OAuth description: 'Bring-your-own OAuth: import externally obtained tokens for a server.' - name: Chatboxes description: >- Read-only access to the chatboxes published from a project: listing, settings, attached servers, and share links. - name: Catalog description: >- Discover the resources the other routes operate on: your account, projects, servers, eval suites, and chat sessions. - name: Tunnels description: >- Relay tunnels that expose local MCP servers through a public URL, registered as first-class project servers (the `mcpjam tunnel` CLI flow). paths: /projects/{projectId}/eval-runs: post: tags: - Eval runs summary: Create an eval run (async) description: >- Creates a suite run from an existing `suiteId` (rerun) and/or inline `tests`, then **detaches execution and responds `202` immediately** with the `runId`. Validation and quota errors surface on this request; poll `GET /eval-runs/{runId}` for progress. The run appears live in the hosted UI Runs tab, tagged `source: "api"`. A bare `suiteId` with no inline tests reruns the suite as configured. Per-organization concurrency is capped (default 2 concurrent runs); exceeding it returns `429` with `details.reason: "CONCURRENT_RUN_LIMIT"`. operationId: createEvalRun parameters: - $ref: '#/components/parameters/projectId' requestBody: required: true content: application/json: schema: $ref: '#/components/schemas/EvalRunCreateRequest' example: suiteName: smoke serverIds: - srv_abc123 tests: - title: echo works runs: 1 model: anthropic/claude-haiku-4.5 provider: anthropic steps: - id: s1 kind: prompt prompt: Use the echo tool to say hi - id: s2 kind: assert assertion: type: toolCalledWith toolName: echo args: args: text: hi responses: '202': description: Run created; execution continues in the background. content: application/json: schema: $ref: '#/components/schemas/EvalRunCreated' example: runId: run_abc123 suiteId: suite_def456 status: running caseUpsert: committed: - name: echo works failed: [] '400': $ref: '#/components/responses/ValidationError' '401': $ref: '#/components/responses/Unauthorized' '403': $ref: '#/components/responses/Forbidden' '404': $ref: '#/components/responses/NotFound' '429': $ref: '#/components/responses/RateLimited' '500': $ref: '#/components/responses/InternalError' '502': $ref: '#/components/responses/ServerUnreachable' '504': $ref: '#/components/responses/Timeout' components: parameters: projectId: name: projectId in: path required: true description: ID of the hosted project that contains the server. schema: type: string schemas: EvalRunCreateRequest: type: object description: >- Two valid shapes: `suiteId` (rerun an existing suite, optionally upserting inline `tests` into it), or `suiteName` + `tests` + `serverIds` (create a new suite and run it). Inline `tests` alone — without a `suiteId` or a `suiteName` — are rejected with `VALIDATION_ERROR`. anyOf: - required: - suiteId - required: - suiteName - tests - serverIds properties: tests: minItems: 1 properties: suiteId: type: string description: >- Existing suite to rerun. A bare `suiteId` with no `tests` reruns the suite exactly as configured. suiteName: type: string description: >- Name for a new suite. Required (non-empty) when no `suiteId` is given. suiteDescription: type: string tests: type: array maxItems: 100 description: Inline test cases to upsert into the suite before running. items: $ref: '#/components/schemas/EvalTestCase' serverIds: type: array minItems: 1 description: >- Servers (by ID) the run connects to. Required when creating a new suite; optional on reruns — when omitted, the run connects the suite's saved server selection (the set its snapshot references). A rerun of a suite with no saved selection is rejected with `VALIDATION_ERROR` (`details.reason: "NO_SAVED_SERVER_SELECTION"`). items: type: string modelApiKeys: type: object description: >- Optional per-provider model API keys (e.g. `{ "anthropic": "sk-ant-…" }`). Falls back to your organization's configured providers when omitted. additionalProperties: type: string notes: type: string passCriteria: type: object properties: minimumPassRate: type: number iterationOverride: type: integer minimum: 1 maximum: 10 description: Override the per-case `runs` count for this run only. additionalProperties: true EvalRunCreated: type: object required: - runId - suiteId - status - caseUpsert properties: runId: type: string suiteId: type: string status: type: string const: running servers: type: array description: >- The servers the run connects to — explicit or derived from the suite's saved selection. `name` is present when known (always, on the derived path). items: type: object required: - id properties: id: type: string name: type: string caseUpsert: type: object description: >- Per-case upsert outcomes for inline `tests`. Partial failures don't abort the run. properties: committed: type: array items: type: object properties: id: type: string name: type: string failed: type: array items: type: object properties: id: type: string name: type: string error: type: string EvalTestCase: type: object description: >- Inline eval test case. The case body is an ordered `steps` array (the unified test model). required: - title - runs - model - provider - steps properties: title: type: string steps: type: array minItems: 1 description: >- Ordered test steps. The first `prompt` step is the case query; `toolCalledWith` asserts are the expected tool calls; a single model-free `toolCall` step is a render-check. items: $ref: '#/components/schemas/EvalTestStep' runs: type: integer minimum: 1 maximum: 10 description: Iterations to execute for this case. model: type: string description: >- Model ID. Hosted-catalog ids use `provider/name` form (e.g. `anthropic/claude-haiku-4.5`) and run on org credits; provider-native ids require a matching `modelApiKeys` entry (BYOK). Unknown models are rejected with VALIDATION_ERROR at create time. provider: type: string description: Model provider, e.g. `anthropic`, `openai`. isNegativeTest: type: boolean description: When `true`, the case passes if NO tools are called. expectedOutput: type: string advancedConfig: type: object description: Optional `system`, `temperature`, `toolChoice` overrides. additionalProperties: true additionalProperties: true Error: type: object required: - code - message properties: code: type: string description: >- Stable, machine-readable error code. New codes may be added over time; treat unknown codes as non-retryable failures unless the HTTP status says otherwise. enum: - UNAUTHORIZED - FORBIDDEN - NOT_FOUND - VALIDATION_ERROR - RATE_LIMITED - FEATURE_NOT_SUPPORTED - SERVER_UNREACHABLE - TIMEOUT - OAUTH_REQUIRED - INTERNAL_ERROR message: type: string description: >- Human-readable description. May change between releases — don't match on it. details: type: object description: Optional, unstructured context bag. additionalProperties: true EvalTestStep: type: object description: >- One authored test step (the unified test model). `kind` discriminates: `prompt` is a user message (model turn); `toolCall` is a deterministic, model-free tool call; `interact` is one pure widget action; `assert` is an assertion (a `Predicate` like `toolCalledWith` / `widgetRendered`, or a DOM `WidgetAssertion`). required: - id - kind properties: id: type: string minLength: 1 kind: type: string enum: - prompt - toolCall - interact - assert prompt: type: string description: 'User message (`kind: prompt`).' serverName: type: string description: 'Server that owns the tool (`kind: toolCall`).' toolName: type: string description: 'Tool name (`kind: toolCall` / `interact`).' arguments: type: object description: 'Tool-call arguments (`kind: toolCall`).' additionalProperties: true action: type: object description: 'Widget action (`kind: interact`).' additionalProperties: true assertion: type: object description: 'Predicate or widget assertion (`kind: assert`).' additionalProperties: true additionalProperties: true responses: ValidationError: description: Malformed body or parameters. content: application/json: schema: $ref: '#/components/schemas/Error' example: code: VALIDATION_ERROR message: Invalid JSON body Unauthorized: description: >- Missing, invalid, revoked, or orphaned key (`UNAUTHORIZED`) — or the **target MCP server** needs an OAuth grant (`OAUTH_REQUIRED`), which is a property of the server, not your key. content: application/json: schema: $ref: '#/components/schemas/Error' examples: badKey: summary: Invalid or revoked key value: code: UNAUTHORIZED message: Invalid API key oauthRequired: summary: Target server needs an OAuth grant value: code: OAUTH_REQUIRED message: Server requires OAuth authorization Forbidden: description: Key is valid but not allowed to do this. content: application/json: schema: $ref: '#/components/schemas/Error' example: code: FORBIDDEN message: You do not have access to this project NotFound: description: Unknown project, server, or resource. content: application/json: schema: $ref: '#/components/schemas/Error' example: code: NOT_FOUND message: Server not found RateLimited: description: >- Per-key rate limit exceeded (60 requests/minute sustained, bursts up to 10). Honor `Retry-After` and back off with jitter. headers: Retry-After: description: Seconds to wait before retrying. schema: type: integer content: application/json: schema: $ref: '#/components/schemas/Error' example: code: RATE_LIMITED message: API key rate limit exceeded. Slow down and retry. InternalError: description: Something failed on MCPJam's side. content: application/json: schema: $ref: '#/components/schemas/Error' example: code: INTERNAL_ERROR message: Unexpected internal error ServerUnreachable: description: Could not connect to the target MCP server. content: application/json: schema: $ref: '#/components/schemas/Error' example: code: SERVER_UNREACHABLE message: Failed to connect to server Timeout: description: The target MCP server connected but didn't respond in time. content: application/json: schema: $ref: '#/components/schemas/Error' example: code: TIMEOUT message: Request to server timed out securitySchemes: bearerAuth: type: http scheme: bearer description: >- MCPJam API key (`sk_…`). Create one at [Settings → API keys](https://app.mcpjam.com/settings/api-keys). Guest sessions cannot use the API, and API keys cannot manage other API keys. ````