Evaluate RAG retrieval, agent tool calls and prompt regressions with deterministic metrics.
Evalsmith-mcp is an MCP integration. Evaluate RAG retrieval, agent tool calls and prompt regressions with deterministic metrics.
The published npm package is evalsmith-mcp, version 0.1.0. Its declared MCP transport is stdio.
The package declares EVALSMITH_JUDGE_BASE_URL: OpenAI-compatible endpoint for the optional LLM judge (default: Ollama on localhost); EVALSMITH_JUDGE_MODEL: Model used by the optional LLM judge; EVALSMITH_JUDGE_API_KEY: API key for the judge endpoint, if it needs one; LOG_LEVEL: pino log level; logs go to stderr.
Check the maintainer’s instructions for account access, authentication and the supported client configuration. Once connected, inspect the tools your client receives and try a small task that matches the server’s stated purpose. The registry description does not include a verified, complete tool list.
View the publisher’s registry entry · Project repository. This profile uses version 0.1.0 from the registry. Registration identifies the publisher namespace; it does not establish that this is a first-party integration for every service it connects.
Browse related developer tools MCP servers for tools that cover the same kind of work.
| Variable | Description | Required |
|---|---|---|
| EVALSMITH_JUDGE_BASE_URL | OpenAI-compatible endpoint for the optional LLM judge (default: Ollama on localhost) | Check documentation |
| EVALSMITH_JUDGE_MODEL | Model used by the optional LLM judge | Check documentation |
| EVALSMITH_JUDGE_API_KEY | API key for the judge endpoint, if it needs one | Check documentation |
| LOG_LEVEL | pino log level; logs go to stderr | Check documentation |
Read, search and edit files inside allowed directories with the Model Context Protocol reference Filesystem server.
Retrieve version-aware library documentation and code examples with Context7 MCP, using resolve-library-id and query-docs.
Use Model Context Protocol with multiple Fireproof JSON document databases
MCP server providing complete integration with Delve Go debugger, supporting all major debugging commands, program analysis, and execution control
【Every star you give feeds a hungry developer's motivation!⭐️】A Model Context Protocol (MCP) server implementation that provides Google Jobs search
Fetches comments from GitHub Pull Requests, including file paths, line ranges, and replies, using a personal access token.