Extracts website content and saves it as markdown files, mapping website structure and links.
Extracts website content and saves it as markdown files, mapping website structure and links. Exposed over MCP by the md webcrawl mcp server, that capability becomes something an assistant can invoke while it works, not something you go and do afterwards.
A Python-based MCP (https://modelcontextprotocol.io/introduction) web crawler for extracting and saving website content.
Everything the assistant can do here goes through one of these:
OUTPUT_PATH — Default output directory for saved filesMAX_CONCURRENT_REQUESTS — Maximum parallel requests (default: 5)REQUEST_TIMEOUT — Request timeout in seconds (default: 30)Debug — It helps to use https://modelcontextprotocol.io/docs/tools/inspector for debuggingInstallation goes through your MCP client rather than a global install: point it at fastmcp on PyPI and it is fetched when the client starts. The copy-paste blocks for Claude Desktop, Claude Code and Cursor are further down this page.
You will need one environment variable: OUTPUT_PATH. Keep credentials in your client's env block or a secrets manager rather than in a file you might commit.
Among the file and storage access options, the useful question is rarely "what can it do" but "what does it cost you to run" — permissions, credentials, and how much of your context its toolset consumes. Md Webcrawl's toolset — OUTPUT_PATH, MAX_CONCURRENT_REQUESTS, REQUEST_TIMEOUT and 1 more — is a fair guide to whether it matches your workflow. It is maintained by jmh108; worth a glance at recent repository activity before you build anything load-bearing on it.
This entry was verified against Md Webcrawl's own documentation before publication; SyncDev keeps the directory reviewed rather than auto-generated.
| Tool | What it does |
|---|---|
| OUTPUT_PATH | Default output directory for saved files |
| MAX_CONCURRENT_REQUESTS | Maximum parallel requests (default: 5) |
| REQUEST_TIMEOUT | Request timeout in seconds (default: 30) |
| Debug | It helps to use https://modelcontextprotocol.io/docs/tools/inspector for debugging |
{
"mcpServers": {
"md-webcrawl": {
"command": "uvx",
"args": ["fastmcp"],
"env": {
"OUTPUT_PATH": "your-value"
}
}
}
}Add to claude_desktop_config.json, then restart Claude Desktop.
| Variable | Description | Required |
|---|---|---|
| OUTPUT_PATH | Filesystem location the server is allowed to use. | Optional |
Scoped local file access — read, write, search and reorganise files in directories you explicitly allow.
Search and read your Drive — Docs, Sheets and files become context your assistant can actually use.
Query, modify and analyse local SQLite databases in conversation — the fastest way to chat with a data file.
Build a programmable telecommunications stack for connecting telephony services with the Internet via a cloud-based utility.
Chat with your second brain — search, read and write vault notes through the Local REST API.
Connects AI models to an Obsidian knowledge base for direct access and manipulation of notes and folders.