Crawl websites, export SEO data, and manage crawls via Screaming Frog SEO Spider.
Most search and retrieval work still happens through a UI a human drives. Screaming Frog SEO Spider MCP server moves it into the conversation instead. Crawl websites, export SEO data, and manage crawls via Screaming Frog SEO Spider.
A headless MCP (Model Context Protocol) server for Screaming Frog SEO Spider. It drives the SF command line and saved crawl database directly, so Claude (or any MCP-compatible client) can run crawls, export crawl data, and analyze the results with the Screaming Frog GUI closed: on your laptop, on a server, or inside scheduled audits and CI pipelines.
The server publishes 9 tools. What each one is for:
sf_check — Verify Screaming Frog is installed, check version and license statuscrawl_site — Start a headless background crawl (see note below)crawl_status — Check progress of a running crawllist_crawls — List all saved crawls with their Database IDsexport_crawl — Export crawl data as CSV files (many export options available)read_crawl_data — Read exported CSV data with pagination, filtering, and column selectionaggregate_crawl_data — Counts and group-by breakdowns over exported data ("how many 404s", "status code distribution") without reading rows into contextdelete_crawl — Permanently delete a crawl from the databasestorage_summary — Show disk usage of SF's crawl storageConfiguration is passed through the environment: SF_CLI_PATH. Treat anything key-shaped as a real credential — scope it to the minimum the server needs, and rotate it if it ever lands in a shared config.
The server ships on PyPI as screaming-frog-mcp, so your MCP client can launch it on demand — there is no separate build step. Add the server block to your client's configuration, restart it, and the tools register themselves.
This sits in the search and retrieval group, where several servers overlap in what they claim to do but differ sharply once you actually set them up. Screaming Frog SEO Spider's toolset — sf_check, crawl_site, crawl_status and 6 more — is a fair guide to whether it matches your workflow. It is maintained by bzsasson; worth a glance at recent repository activity before you build anything load-bearing on it.
We check each listing at SyncDev against the project's documentation before it goes live — if something here drifts out of date, it is a bug worth reporting.
| Tool | What it does |
|---|---|
| sf_check | Verify Screaming Frog is installed, check version and license status |
| crawl_site | Start a headless background crawl (see note below) |
| crawl_status | Check progress of a running crawl |
| list_crawls | List all saved crawls with their Database IDs |
| export_crawl | Export crawl data as CSV files (many export options available) |
| read_crawl_data | Read exported CSV data with pagination, filtering, and column selection |
| aggregate_crawl_data | Counts and group-by breakdowns over exported data ("how many 404s", "status code distribution") without reading rows into context |
| delete_crawl | Permanently delete a crawl from the database |
| storage_summary | Show disk usage of SF's crawl storage |
{
"mcpServers": {
"screaming-frog": {
"command": "uvx",
"args": ["screaming-frog-mcp"],
"env": {
"SF_CLI_PATH": "your-value"
}
}
}
}Add to claude_desktop_config.json, then restart Claude Desktop.
| Variable | Description | Required |
|---|---|---|
| SF_CLI_PATH | Filesystem location the server is allowed to use. | Optional |
The simplest web tool that matters — fetch any URL and get model-ready markdown back.
Industrial-strength web extraction — render, scrape, crawl and search entire sites into clean markdown.
Puppeteer-powered browser control that drives pages from the accessibility tree instead of pixels.
Search built for AI, not humans — semantic web search that returns model-ready content, plus code context.
Answers, not links — delegate questions to Perplexity's search-grounded models and get cited responses back.
Put 6,000+ pre-built scrapers at your assistant's fingertips through one MCP endpoint.