SSyncDev

Evalsmith-mcp Server

Evaluate RAG retrieval, agent tool calls and prompt regressions with deterministic metrics.

Local serverstdio

What is the Evalsmith-mcp Server?

Evalsmith-mcp is an MCP integration. Evaluate RAG retrieval, agent tool calls and prompt regressions with deterministic metrics.

Packages and local setup

The published npm package is evalsmith-mcp, version 0.1.0. Its declared MCP transport is stdio.

The package declares EVALSMITH_JUDGE_BASE_URL: OpenAI-compatible endpoint for the optional LLM judge (default: Ollama on localhost); EVALSMITH_JUDGE_MODEL: Model used by the optional LLM judge; EVALSMITH_JUDGE_API_KEY: API key for the judge endpoint, if it needs one; LOG_LEVEL: pino log level; logs go to stderr.

Before connecting

Check the maintainer’s instructions for account access, authentication and the supported client configuration. Once connected, inspect the tools your client receives and try a small task that matches the server’s stated purpose. The registry description does not include a verified, complete tool list.

Project and source

View the publisher’s registry entry · Project repository. This profile uses version 0.1.0 from the registry. Registration identifies the publisher namespace; it does not establish that this is a first-party integration for every service it connects.

Browse related developer tools MCP servers for tools that cover the same kind of work.

Configuration

VariableDescriptionRequired
EVALSMITH_JUDGE_BASE_URLOpenAI-compatible endpoint for the optional LLM judge (default: Ollama on localhost)Check documentation
EVALSMITH_JUDGE_MODELModel used by the optional LLM judgeCheck documentation
EVALSMITH_JUDGE_API_KEYAPI key for the judge endpoint, if it needs oneCheck documentation
LOG_LEVELpino log level; logs go to stderrCheck documentation