Crawl and analyse websites for SEO errors using Crawlee with SQLite storage
Crawl and analyse websites for SEO errors using Crawlee with SQLite storage
I wanted to build on my experience working with the MCP protocol SDK to see just how far we can extend an AI assistant's capabilities. I decided that I'd quite like to build a crawler to check my site's "technical SEO" health and came across Crawlee - which seemed like the ideal library to base the crawl component of my MCP.
Compare MCP servers by category, or browse hosted MCP servers if you prefer an endpoint over a local process.
Check the linked publisher documentation and your connected client's tool list. Names and availability can vary with the server version and account permissions.
| Tool | What it does |
|---|---|
| run_seo_audit | Crawl a website and extract comprehensive SEO data into SQLite. |
| analyze_seo | |
| query_seo_data | |
| list_seo_queries |
{
"mcpServers": {
"seo-crawler-mcp": {
"command": "npx",
"args": ["-y", "@houtini/seo-crawler-mcp"],
"env": {
"OUTPUT_DIR": "C:\\seo-audits"
}
}
}
}Configuration as documented by the project. Restart the client after saving.
| Variable | Description | Required |
|---|---|---|
| OUTPUT_DIR | Filesystem location the server is allowed to use. | Optional |
| DEBUG | Configuration value read at startup. | Optional |
Retrieve public web pages as readable Markdown with the reference Fetch MCP server, including pagination for longer responses.
Use Firecrawl MCP to retrieve website content and extract information for research through Firecrawl’s API-backed web tools.
The original Chromium automation reference server — simple, screenshot-driven browser control.
Puppeteer-powered browser control that drives pages from the accessibility tree instead of pixels.
Automates browsers and APIs, enabling Large Language Models (LLMs) to interact with web pages for tasks like screenshot capture and JavaScript
Search built for AI, not humans — semantic web search that returns model-ready content, plus code context.