A Model Context Protocol (MCP) server that provides a web crawling and content extraction tool for AI assistants such as Claude Desktop, Cursor, and other MCP-compatible clients.
Web Crawler MCP Server is an MCP project maintained in the jonathanhsuhh/web-crawler-mcp-server repository. The maintainer describes its purpose as “A Model Context Protocol (MCP) server that provides a web crawling and content extraction tool for AI assistants such as Claude Desktop, Cursor, and other…”. Read the project documentation for the complete scope.
The project provides a local client configuration using node. Follow the repository’s build and installation steps before copying it, and replace example paths with the paths on your own computer.
Project documentation · Repository · Discovery listing. This profile records the public documentation checked on 2026-10-08.
Check the linked publisher documentation and your connected client's tool list. Names and availability can vary with the server version and account permissions.
| Tool | What it does |
|---|---|
| Input: | url (string, required): The URL to extract content from. |
{
"mcpServers": {
"web-crawler-mcp": {
"command": "node",
"args": [
"<absolute-path-to>/server/web_crawler/build/index.js"
]
}
}
}From the project documentation. Replace placeholders and example paths before connecting.
Retrieve public web pages as readable Markdown with the reference Fetch MCP server, including pagination for longer responses.
Use Firecrawl MCP to retrieve website content and extract information for research through Firecrawl’s API-backed web tools.
A Model Context Protocol (MCP) server that provides web search capabilities through DuckDuckGo, with additional features for content fetching and
Official MCP server for the Massive Web Render API. Give your AI agents real-time web access — fetch any URL, search Google, query AI chatbots — with JS rendering, captcha solving, and 195+ country…
MCP server for the WebberSites x402 Data API — 45 pay-per-call tools for AI agents: web scraping, document extraction (PDF/DOCX/CSV), SEO/schema/accessibility audits (single page or whole site), deterministic code lint…
Eval-backed tool discovery for AI agents, on the auxiliar.ai web-access gateway — one API key for 24 search, scraping, browser-automation and voice APIs, upstream keys injected server-side, usage at…