Llmprobe MCP Server

llmprobe is a CLI tool for LLM serving reliability. It probes hosted APIs or OpenAI-compatible inference servers, then reports the metrics that matter for production user experience: time to first token (TTFT), total latency, generation. The llmprobe mcp server wraps that behind the Model Context Protocol, so an assistant can use it through 4 defined tools rather than through you.

Adding it to your client

Being a remote server, there is no local install. You register the endpoint with your client, authorise it once, and the tools appear.

Its toolset

Everything the assistant can do here goes through one of these:

• probe_all — Probe all configured endpoints from probes.yml. Returns TTFT, latency, throughput, and health status for every model. Accepts an optional config
• probe_model — Probe a single model without a config file. Requires provider, model, and api_key_env. Supports optional base_url for OpenAI-compatible endpoints and
• list_providers — List all providers and models in the config file with their thresholds. Use this to discover available models before probing
• get_config — Return the full parsed configuration including defaults, providers, models, and thresholds

Configuration

You will need 8 environment variables: OPENAI_API_KEY, ANTHROPIC_API_KEY, AZURE_OPENAI_API_KEY, AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, GROQ_API_KEY, DEEPSEEK_API_KEY, TOGETHER_API_KEY. The server will not start without them, which is usually why the tools fail to appear on a first run. Keep credentials in your client's env block or a secrets manager rather than in a file you might commit.

Caveats

• Your data travels to the provider's service, so the usual questions apply about what you send and what they retain.
• Missing credentials fail quietly in some clients — if no tools show up, check the environment block first.
• MCP clients confirm each tool call by default. Leave that on until you have watched what the llmprobe mcp server does with a few real requests.

When to reach for it

This sits in the AI and media services group, where several servers overlap in what they claim to do but differ sharply once you actually set them up. Llmprobe's toolset — probe_all, probe_model, list_providers and 1 more — is a fair guide to whether it matches your workflow. It is maintained by jwrede; worth a glance at recent repository activity before you build anything load-bearing on it.

We check each listing at SyncDev against the project's documentation before it goes live — if something here drifts out of date, it is a bug worth reporting.