<em>Web Crawling and RAG Capabilities for AI Agents and AI Coding Assistants</em>
Web Crawling and RAG Capabilities for AI Agents and AI Coding Assistants
This MCP server provides tools that enable AI agents to crawl websites, store content in a vector database (Supabase), and perform RAG over the crawled content. It follows the best practices for building MCP servers based on the Mem0 MCP server template I provided on my channel previously.
Compare MCP servers by category, or browse hosted MCP servers if you prefer an endpoint over a local process.
{
"mcpServers": {
"crawl4ai-rag": {
"transport": "sse",
"url": "http://localhost:8051/sse"
}
}
}Configuration as documented by the project. Restart the client after saving.
| Variable | Description | Required |
|---|---|---|
| TRANSPORT | Configuration value read at startup. | Optional |
| OPENAI_API_KEY | Credential the server authenticates with. | Yes |
| SUPABASE_URL | Endpoint or connection string the server talks to. | Yes |
| SUPABASE_SERVICE_KEY | Credential the server authenticates with. | Yes |
| USE_KNOWLEDGE_GRAPH | Configuration value read at startup. | Optional |
| NEO4J_URI | Configuration value read at startup. | Optional |
| NEO4J_USER | Configuration value read at startup. | Optional |
| NEO4J_PASSWORD | Configuration value read at startup. | Optional |
Retrieve public web pages as readable Markdown with the reference Fetch MCP server, including pagination for longer responses.
Use Firecrawl MCP to retrieve website content and extract information for research through Firecrawl’s API-backed web tools.
The original Chromium automation reference server — simple, screenshot-driven browser control.
Puppeteer-powered browser control that drives pages from the accessibility tree instead of pixels.
Automates browsers and APIs, enabling Large Language Models (LLMs) to interact with web pages for tasks like screenshot capture and JavaScript
Search built for AI, not humans — semantic web search that returns model-ready content, plus code context.