Where does the crawled data and cache reside?

All Markdown files are stored in the `./storage/` directory. The server's processed chunks and embeddings are cached in `storage/document_chunks_cache.pkl`. The cache is invalidated when any `.md` file changes.

What transport and authentication does the MCP server use?

The server runs via `stdio` transport (no HTTP). No authentication is mentioned in the README; it relies on the local execution environment.

SSyncDev

Start a project

Documentation Crawler & MCP Server

This project provides a toolset to crawl websites wikis, tool/library documentions and generate Markdown documentation, and make that documentation

Local serverstdioPython

What is the Documentation Crawler & MCP Server?

This project provides a toolset to crawl websites wikis, tool/library documentions and generate Markdown documentation, and make that documentation searchable via a

What the server does

Web Crawler (crawler_cli) —
Crawls websites starting from a given URL using crawl4ai
Configurable crawl depth, URL patterns (include/exclude), content types, etc
Optional cleaning of HTML before Markdown conversion (removes nav links, headers, footers)
Generates a single, consolidated Markdown file from crawled content
Saves output to ./storage/ by default

More MCP options

Compare MCP servers by category, or browse hosted MCP servers if you prefer an endpoint over a local process.

Maintainer setup documentation.

Frequently asked questions

Python 3 with `uv`, and key libraries: `crawl4ai`, `fastmcp`, `sentence-transformers`, `torch`. A compatible GPU (CUDA or Apple Silicon/MPS) speeds up embedding generation, but CPU works.