TheCrawler — Apify actor wrapper around the standalone `thecrawler` npm package. All scraping logic lives in the standalone engine; this is a thin
TheCrawler — Apify actor wrapper around the standalone thecrawler npm package. All scraping logic lives in the standalone engine; this is a thin charging +
Scrape web pages, run LLM-powered structured extraction, or diagnose whether URLs are ready for a built-in extraction contract before spending LLM tokens. Open source engine (AGPL-3.0). $0.005 per successfully scraped page on Apify.
Compare MCP servers by category, or browse hosted MCP servers if you prefer an endpoint over a local process.
{
"mcpServers": {
"thecrawler": {
"command": "npx",
"args": ["-y", "npm"],
"env": {
"THECRAWLER_LLM_API_KEY": "your-value",
"THECRAWLER_API_KEY": "your-value"
}
}
}
}Add to claude_desktop_config.json, then restart Claude Desktop.
| Variable | Description | Required |
|---|---|---|
| THECRAWLER_LLM_API_KEY | Credential the server authenticates with. | Yes |
| THECRAWLER_API_KEY | Credential the server authenticates with. | Yes |
Retrieve public web pages as readable Markdown with the reference Fetch MCP server, including pagination for longer responses.
Use Firecrawl MCP to retrieve website content and extract information for research through Firecrawl’s API-backed web tools.
SEO audits and AI crawler checks. Honest findings only; no invented scores.
Fetch URLs and return clean Markdown for RAG — nav, ads and boilerplate stripped.
MCP server for Google Web Search using Gemini API
Detects a company CRM, sequencer, and marketing automation from its public website. Clay-ready.