MCP server tailored to connecting web crawler data and archives
MCP server tailored to connecting web crawler data and archives
Advanced search and retrieval for web crawler data. With mcp-server-webcrawl, your AI client filters and analyzes web content under your direction or autonomously. The server includes a fulltext search interface with boolean support, and resource filtering by type, HTTP status, and more.
Once MCP is connected, these are the calls the assistant has available:
Multi — crawler compatiblePrompt — Downloadprivacy — fulltext single keyword matchboundar — fulltext wildcard matches results starting with boundar (boundary, boundaries)Field — Descriptionurl — resource URLtype — enumerated list of types (see types table)size — file size in bytesstatus — HTTP response codesheaders — HTTP response headerscontent — HTTP body—HTML, CSS, JS, and morecreated — on requestprivacy — fulltext single keyword matchboundar — fulltext wildcard matches results starting with boundar (boundary, boundaries)url — resource URLtype — enumerated list of types (see types table)size — file size in bytesstatus — HTTP response codesheaders — HTTP response headerscontent — HTTP body—HTML, CSS, JS, and morecreated — on requestmodified — on requestCompare MCP servers by category, or browse hosted MCP servers if you prefer an endpoint over a local process.
Check the linked publisher documentation and your connected client's tool list. Names and availability can vary with the server version and account permissions.
| Tool | What it does |
|---|---|
| privacy | fulltext single keyword match |
| boundar | fulltext wildcard matches results starting with *boundar* (boundary, boundaries) |
| url | resource URL |
| type | enumerated list of types (see types table) |
| size | file size in bytes |
| status | HTTP response codes |
| headers | HTTP response headers |
| content | HTTP body—HTML, CSS, JS, and more |
| created | on request |
| modified | on request |
{
"mcpServers": {
"server-webcrawl": {
"command": "uvx",
"args": ["mcp-server-webcrawl"]
}
}
}Add to claude_desktop_config.json, then restart Claude Desktop.
Retrieve public web pages as readable Markdown with the reference Fetch MCP server, including pagination for longer responses.
Use Firecrawl MCP to retrieve website content and extract information for research through Firecrawl’s API-backed web tools.
The original Chromium automation reference server — simple, screenshot-driven browser control.
Puppeteer-powered browser control that drives pages from the accessibility tree instead of pixels.
Automates browsers and APIs, enabling Large Language Models (LLMs) to interact with web pages for tasks like screenshot capture and JavaScript
Search built for AI, not humans — semantic web search that returns model-ready content, plus code context.