CrawlForge MCP is a production-ready MCP server with 18 web scraping tools for AI agents. It gives Claude, Cursor, and any MCP-compatible client the
If you already use CrawlForge MCP, the crawlforge mcp mcp server is the piece that lets your assistant work with it directly. CrawlForge MCP is a production-ready MCP server with 18 web scraping tools for AI agents. It gives Claude, Cursor, and any MCP-compatible client the ability to fetch URLs, extract structured data with CSS/XPath selectors, run deep.
27 web scraping, crawling, deep-research & autonomous-extraction tools for Claude, Cursor & any MCP client.
The toolset is worth reading before you wire it up, because it tells you what the integration is really for:
Auto — configure Claude Code and Cursor** (if installed)fetch_url — 1extract_text — 1extract_links — 1extract_metadata — 1scrape_template — 1list_ollama_models — 1get_batch_results — 1scrape — 2scrape_structured — 2extract_content — 2map_site — 2Configuration is passed through the environment: CRAWLFORGE_API_KEY, CRAWLFORGE_API_URL, OLLAMA_BASE_URL, OLLAMA_DEFAULT_MODEL, OPENAI_API_KEY, ANTHROPIC_API_KEY, CRAWLFORGE_TOOLS, CRAWLFORGE_TOOL_GROUPS. Treat anything key-shaped as a real credential — scope it to the minimum the server needs, and rotate it if it ever lands in a shared config.
Because this one is hosted, setup is mostly authentication — you point your client at the endpoint and approve access. Nothing runs on your machine, so there is no runtime to keep patched.
Among the search and retrieval options, the useful question is rarely "what can it do" but "what does it cost you to run" — permissions, credentials, and how much of your context its toolset consumes. CrawlForge MCP's toolset — Auto, fetch_url, extract_text and 11 more — is a fair guide to whether it matches your workflow.
SyncDev reviews every entry in this directory against the project's own documentation before publishing, and revisits them as servers change.
| Tool | What it does |
|---|---|
| Auto | configure Claude Code and Cursor** (if installed) |
| fetch_url | 1 |
| extract_text | 1 |
| extract_links | 1 |
| extract_metadata | 1 |
| scrape_template | 1 |
| list_ollama_models | 1 |
| get_batch_results | 1 |
| scrape | 2 |
| scrape_structured | 2 |
| extract_content | 2 |
| map_site | 2 |
| process_document | 2 |
| localization | 2 |
{
"mcpServers": {
"crawlforge-mcp-git": {
"command": "npx",
"args": ["-y", "crawlforge-setup"],
"env": {
"CRAWLFORGE_API_KEY": "your-value",
"CRAWLFORGE_API_URL": "your-value",
"OLLAMA_BASE_URL": "your-value",
"OLLAMA_DEFAULT_MODEL": "your-value",
"OPENAI_API_KEY": "your-value",
"ANTHROPIC_API_KEY": "your-value",
"CRAWLFORGE_TOOLS": "your-value",
"CRAWLFORGE_TOOL_GROUPS": "your-value"
}
}
}
}Add to claude_desktop_config.json, then restart Claude Desktop.
| Variable | Description | Required |
|---|---|---|
| CRAWLFORGE_API_KEY | Credential the server authenticates with. | Yes |
| CRAWLFORGE_API_URL | Endpoint or connection string the server talks to. | Yes |
| OLLAMA_BASE_URL | Endpoint or connection string the server talks to. | Yes |
| OLLAMA_DEFAULT_MODEL | Configuration value read at startup. | Optional |
| OPENAI_API_KEY | Credential the server authenticates with. | Yes |
| ANTHROPIC_API_KEY | Credential the server authenticates with. | Yes |
| CRAWLFORGE_TOOLS | Configuration value read at startup. | Optional |
| CRAWLFORGE_TOOL_GROUPS | Configuration value read at startup. | Optional |
The simplest web tool that matters — fetch any URL and get model-ready markdown back.
Industrial-strength web extraction — render, scrape, crawl and search entire sites into clean markdown.
Puppeteer-powered browser control that drives pages from the accessibility tree instead of pixels.
Search built for AI, not humans — semantic web search that returns model-ready content, plus code context.
Answers, not links — delegate questions to Perplexity's search-grounded models and get cited responses back.
Put 6,000+ pre-built scrapers at your assistant's fingertips through one MCP endpoint.