MCP Eval Runner MCP Server

MCP Eval Runner MCP server is a locally run integration for AI assistants that speak the Model Context Protocol. A standardized testing harness for MCP servers and agent workflows.

What you get

A standardized testing harness for MCP servers and agent workflows. Define test cases as YAML fixtures (steps → expected tool calls → expected outputs), run regression suites directly from your MCP client, and get pass/fail results with diffs — without leaving Claude Code or Cursor.

• YAML fixtures — Test cases are plain files in version control — diffable, reviewable, and shareable
• Two execution modes — Live mode spawns a real MCP server and calls tools via stdio; simulation mode runs assertions against expected_output without a server
• Composable assertions — Combine output_contains, output_not_contains, output_equals, output_matches, schema_match, tool_called, and latency_under per step
• Step output piping — Reference a previous step's output in downstream inputs via {{steps.<step_id>.output}}
• Regression reports — Compare the current run to any past run and surface what changed
• Watch mode — Automatically reruns the affected fixture when files change

What the assistant can call

Once MCP Eval Runner is connected, these are the calls the assistant has available:

• Running — The Running tool exposed by this server
• Authoring — The Authoring tool exposed by this server
• Reporting — The Reporting tool exposed by this server
• Operations — The Operations tool exposed by this server

Configuration and credentials

• Node.js v22.5.0 or newer. - npm.

Setting it up

Installation goes through your MCP client rather than a global install: point it at @modelcontextprotocol/inspector on npm and it is fetched when the client starts. The copy-paste blocks for Claude Desktop, Claude Code and Cursor are further down this page.

Choosing this one

This sits in the developer tooling group, where several servers overlap in what they claim to do but differ sharply once you actually set them up. MCP Eval Runner's toolset — Running, Authoring, Reporting and 1 more — is a fair guide to whether it matches your workflow. It is maintained by dbsectrainer; worth a glance at recent repository activity before you build anything load-bearing on it.

This entry was verified against MCP Eval Runner's own documentation before publication; SyncDev keeps the directory reviewed rather than auto-generated.

Before you rely on it

• It runs with your machine's permissions. That is convenient and also the reason to think about what you point it at before you approve a tool call.
• MCP clients confirm each tool call by default. Leave that on until you have watched what the mcp eval runner mcp server does with a few real requests.