MCP server tailored to connecting web crawler data and archives
MCP MCP server exists for a simple reason — assistants are far more useful when they can act on MCP directly instead of describing what you should do. MCP server tailored to connecting web crawler data and archives.
Advanced search and retrieval for web crawler data. With mcp-server-webcrawl, your AI client filters and analyzes web content under your direction or autonomously. The server includes a fulltext search interface with boolean support, and resource filtering by type, HTTP status, and more.
Once MCP is connected, these are the calls the assistant has available:
Multi — crawler compatiblePrompt — Downloadprivacy — fulltext single keyword matchboundar — fulltext wildcard matches results starting with boundar (boundary, boundaries)Field — Descriptionurl — resource URLtype — enumerated list of types (see types table)size — file size in bytesstatus — HTTP response codesheaders — HTTP response headerscontent — HTTP body—HTML, CSS, JS, and morecreated — on requestThe server ships on PyPI as mcp-server-webcrawl, so your MCP client can launch it on demand — there is no separate build step. Add the server block to your client's configuration, restart it, and the tools register themselves.
This sits in the search and retrieval group, where several servers overlap in what they claim to do but differ sharply once you actually set them up. MCP's toolset — Multi, Prompt, privacy and 11 more — is a fair guide to whether it matches your workflow. It is maintained by pragmar; worth a glance at recent repository activity before you build anything load-bearing on it.
This entry was verified against MCP's own documentation before publication; SyncDev keeps the directory reviewed rather than auto-generated.
| Tool | What it does |
|---|---|
| Multi | crawler compatible |
| Prompt | Download |
| privacy | fulltext single keyword match |
| boundar | fulltext wildcard matches results starting with *boundar* (boundary, boundaries) |
| Field | Description |
| url | resource URL |
| type | enumerated list of types (see types table) |
| size | file size in bytes |
| status | HTTP response codes |
| headers | HTTP response headers |
| content | HTTP body—HTML, CSS, JS, and more |
| created | on request |
| modified | on request |
| Type | Description |
{
"mcpServers": {
"server-webcrawl": {
"command": "uvx",
"args": ["mcp-server-webcrawl"]
}
}
}Add to claude_desktop_config.json, then restart Claude Desktop.
The simplest web tool that matters — fetch any URL and get model-ready markdown back.
Industrial-strength web extraction — render, scrape, crawl and search entire sites into clean markdown.
Puppeteer-powered browser control that drives pages from the accessibility tree instead of pixels.
Search built for AI, not humans — semantic web search that returns model-ready content, plus code context.
Answers, not links — delegate questions to Perplexity's search-grounded models and get cited responses back.
Put 6,000+ pre-built scrapers at your assistant's fingertips through one MCP endpoint.