SSyncDev

MCP Eval Runner MCP Server

A standardized testing harness for MCP servers and agent workflows

Local serverstdioGo

What is the MCP Eval Runner MCP Server?

A standardized testing harness for MCP servers and agent workflows

What you get

A standardized testing harness for MCP servers and agent workflows. Define test cases as YAML fixtures (steps → expected tool calls → expected outputs), run regression suites directly from your MCP client, and get pass/fail results with diffs — without leaving Claude Code or Cursor.

  • YAML fixtures — Test cases are plain files in version control — diffable, reviewable, and shareable
  • Two execution modes — Live mode spawns a real MCP server and calls tools via stdio; simulation mode runs assertions against expected_output without a server
  • Composable assertions — Combine output_contains, output_not_contains, output_equals, output_matches, schema_match, tool_called, and latency_under per step
  • Step output piping — Reference a previous step's output in downstream inputs via {{steps.<step_id>.output}}
  • Regression reports — Compare the current run to any past run and surface what changed
  • Watch mode — Automatically reruns the affected fixture when files change

Configuration and credentials

  • Node.js v22.5.0 or newer. - npm.

More MCP options

Compare MCP servers by category, or browse hosted MCP servers if you prefer an endpoint over a local process.

Maintainer setup documentation.

How to install the MCP Eval Runner MCP Server

{
  "mcpServers": {
    "eval-runner": {
      "command": "npx",
      "args": ["-y", "mcp-eval-runner@latest"]
    }
  }
}

Configuration as documented by the project. Restart the client after saving.

Configuration

  • Node.js v22.5.0 or newer. - npm.

Example prompts to try

  • “Use MCP Eval Runner to Running.”
  • “Use MCP Eval Runner to Authoring.”
  • “Use MCP Eval Runner to Reporting.”

Frequently asked questions

A standardized testing harness for MCP servers and agent workflows Confirm the tools offered by your installed version in the connected client. Tool availability and permissions depend on the server and account configuration.

Alternatives to the MCP Eval Runner MCP Server

Filesystem

Official

Read, search and edit files inside allowed directories with the Model Context Protocol reference Filesystem server.

File Systems & StorageTypeScript★ 63.0k

Context7

Official

Retrieve version-aware library documentation and code examples with Context7 MCP, using resolve-library-id and query-docs.

Documentation & API ReferencesRemoteTypeScript★ 32.0k

Mcp-seedream-pro

MCP server for ByteDance Seedream AI image generation

AI & ML ServicesRemote

Router-mcp

Run the Intelliverse platform from any agent: apps, API keys, knowledge bases, email, media, chat.

Developer Tools

Pydantic Rpc

Below is an example of a simple gRPC service that exposes a [PydanticAI](https://ai.pydantic.dev/) agent:

Developer ToolsPython

Shopscout-mcp

Let your agent shop and compare: Shopify catalog search, offer ranking, shipping plans. $0.01/call.

Payments & Commerce