mcp-server-spider: A spider MCP server

Please note that mcp-server-spider is currently in early develpoment. There might be bugs and features added in the future.

Local server

What is the mcp-server-spider: A spider MCP server?

mcp-server-spider: A spider MCP server is an MCP project maintained in the georgels/mcp-server-spider repository. The maintainer describes its purpose as “Please note that mcp-server-spider is currently in early develpoment. There might be bugs and features added in the future”. Read the project documentation for the complete scope.

Source

Project documentation · Repository · Discovery listing. This profile records the public documentation checked on 2026-10-08.

Recorded tools

Check the linked publisher documentation and your connected client's tool list. Names and availability can vary with the server version and account permissions.

ToolWhat it does
Input:url: The url to crawl
headersAdditional headers passed along with crawl requests
user_agentUser agent to use for the crawl requests
depthThe depth of link traversal
blacklistA list of regural expression to blacklist URLs from the crawling process
whitelistA list of regular expression to whitelist URLS from the crawling process
respect_robots_txtWhether to respect robots.txt file
accept_invalid_certsWhether to accept invalid certifcates or not
ReturnsA list of JSON objects (as a string) that contain the url, links and content of each page discovered