MCP-compliant web scraper with advanced cookie consent handling
MCP-compliant web scraper with advanced cookie consent handling. The mcp web scraper mcp server wraps that behind the Model Context Protocol, so an assistant can use it through 7 defined tools rather than through you.
A production-ready global content extraction platform with ML-powered automation, international site support, and intelligent optimization. Features complete browser automation with 29 tools, 21+ supported sites across 4 regions, 6 content platforms with specialized optimization, and persistent SQLite caching with cross-session learning. Built with TypeScript using the official MCP TypeScript SDK and Playwright.
tsx on npm is all you need. Most clients run it directly, so configuration is a few lines and a restart.
Everything the assistant can do here goes through one of these:
scrape_article_content — Extract article content with intelligent cookie consent handlingget_page_screenshot — The get_page_screenshot tool exposed by this serverhandle_cookie_consent — The handle_cookie_consent tool exposed by this servermanage_tabs — The manage_tabs tool exposed by this servermonitor_network — The monitor_network tool exposed by this serverdrag_drop — The drag_drop tool exposed by this servernavigate_history — The navigate_history tool exposed by this serverYou will need 2 environment variables: BROWSER_POOL_SIZE, DEBUG_LOGGING. Keep credentials in your client's env block or a secrets manager rather than in a file you might commit.
This sits in the search and retrieval group, where several servers overlap in what they claim to do but differ sharply once you actually set them up. MCP Web Scraper's toolset — scrape_article_content, get_page_screenshot, handle_cookie_consent and 4 more — is a fair guide to whether it matches your workflow. It is maintained by descoped; worth a glance at recent repository activity before you build anything load-bearing on it.
We check each listing at SyncDev against the project's documentation before it goes live — if something here drifts out of date, it is a bug worth reporting.
| Tool | What it does |
|---|---|
| scrape_article_content | Extract article content with intelligent cookie consent handling. |
| get_page_screenshot | The get_page_screenshot tool exposed by this server. |
| handle_cookie_consent | The handle_cookie_consent tool exposed by this server. |
| manage_tabs | The manage_tabs tool exposed by this server. |
| monitor_network | The monitor_network tool exposed by this server. |
| drag_drop | The drag_drop tool exposed by this server. |
| navigate_history | The navigate_history tool exposed by this server. |
{
"mcpServers": {
"web-scraper": {
"command": "npx",
"args": ["-y", "tsx"],
"env": {
"BROWSER_POOL_SIZE": "your-value",
"DEBUG_LOGGING": "your-value"
}
}
}
}Add to claude_desktop_config.json, then restart Claude Desktop.
| Variable | Description | Required |
|---|---|---|
| BROWSER_POOL_SIZE | Configuration value read at startup. | Optional |
| DEBUG_LOGGING | Configuration value read at startup. | Optional |
The simplest web tool that matters — fetch any URL and get model-ready markdown back.
Industrial-strength web extraction — render, scrape, crawl and search entire sites into clean markdown.
Puppeteer-powered browser control that drives pages from the accessibility tree instead of pixels.
Search built for AI, not humans — semantic web search that returns model-ready content, plus code context.
Answers, not links — delegate questions to Perplexity's search-grounded models and get cited responses back.
Put 6,000+ pre-built scrapers at your assistant's fingertips through one MCP endpoint.