Download website to a local directory (including all css, images, js, etc.)
Most search and retrieval work still happens through a UI a human drives. Website MCP server moves it into the conversation instead. Download website to a local directory (including all css, images, js, etc.).
Installation goes through your MCP client rather than a global install: point it at website-scraper on npm and it is fetched when the client starts. The copy-paste blocks for Claude Desktop, Claude Code and Cursor are further down this page.
The server publishes 14 tools. What each one is for:
urls — Array of objects which contain urls to download and filenames for them. Required. javascript scrape({ urls: [ 'http://nodejs.org/', // Will bedirectory — String, absolute path to directory where downloaded files will be saved. Directory should not exist. It will be created by scraper. Requiredsources — Array of objects to download, specifies selectors and attribute values to select files for downloading. By default scraper tries to download allrecursive — Boolean, if true scraper will follow hyperlinks in html files. Don't forget to set maxRecursiveDepth to avoid infinite downloading. Defaults to falsemaxRecursiveDepth — Positive number, maximum allowed depth for hyperlinks. Other dependencies will be saved regardless of their depth. Defaults to null - no maximummaxDepth — Positive number, maximum allowed depth for all dependencies. Defaults to null - no maximum depth set. In most of cases you needrequest — Object, custom options for http module got which is used inside website-scraper. Allows to setsubdirectories — Array of objects, specifies subdirectories for file extensions. If null all files will be saved to directory. javascript /* Separate files intodefaultFilename — The defaultFilename tool exposed by this serverprettifyUrls — Boolean, whether urls should be 'prettified', by having the defaultFilename removed. Defaults to falseignoreErrors — Boolean, if true scraper will continue downloading resources after error occurred, if false - scraper will finish process and return error. DefaultsurlFilter — Function which is called for each url to check whether it should be scraped. Defaults to null - no url filter will be applied. javascript // Links toConfiguration is passed through the environment: DEBUG. Treat anything key-shaped as a real credential — scope it to the minimum the server needs, and rotate it if it ever lands in a shared config.
Among the search and retrieval options, the useful question is rarely "what can it do" but "what does it cost you to run" — permissions, credentials, and how much of your context its toolset consumes. Website's toolset — urls, directory, sources and 11 more — is a fair guide to whether it matches your workflow. It is maintained by GitHub Actions; worth a glance at recent repository activity before you build anything load-bearing on it.
This entry was verified against Website's own documentation before publication; SyncDev keeps the directory reviewed rather than auto-generated.
| Tool | What it does |
|---|---|
| urls | Array of objects which contain urls to download and filenames for them. **_Required_**. javascript scrape({ urls: [ 'http://nodejs.org/', // Will be saved with default filename 'index.html' {url: 'http://nodejs.org/about |
| directory | String, absolute path to directory where downloaded files will be saved. Directory should not exist. It will be created by scraper. **_Required_**. How to download website to existing directory and why it's not supported |
| sources | Array of objects to download, specifies selectors and attribute values to select files for downloading. By default scraper tries to download all possible resources. Scraper uses cheerio to select html elements so selecto |
| recursive | Boolean, if true scraper will follow hyperlinks in html files. Don't forget to set maxRecursiveDepth to avoid infinite downloading. Defaults to false. |
| maxRecursiveDepth | Positive number, maximum allowed depth for hyperlinks. Other dependencies will be saved regardless of their depth. Defaults to null - no maximum recursive depth set. |
| maxDepth | Positive number, maximum allowed depth for all dependencies. Defaults to null - no maximum depth set. In most of cases you need [maxRecursiveDepth](#maxRecursiveDepth) instead of this option. |
| request | Object, custom options for http module [got](https://github.com/sindresorhus/got#options) which is used inside website-scraper. Allows to set retries, cookies, userAgent, encoding, etc. javascript // use same request opt |
| subdirectories | Array of objects, specifies subdirectories for file extensions. If null all files will be saved to directory. javascript /* Separate files into directories: - img for .jpg, .png, .svg (full path /path/to/save/img) - js f |
| defaultFilename | The defaultFilename tool exposed by this server. |
| prettifyUrls | Boolean, whether urls should be 'prettified', by having the defaultFilename removed. Defaults to false. |
| ignoreErrors | Boolean, if true scraper will continue downloading resources after error occurred, if false - scraper will finish process and return error. Defaults to false. |
| urlFilter | Function which is called for each url to check whether it should be scraped. Defaults to null - no url filter will be applied. javascript // Links to other websites are filtered out by the urlFilter scrape({ urls: ['http |
| filenameGenerator | String (name of the bundled filenameGenerator). Filename generator determines path in file system where the resource will be saved. |
| requestConcurrency | Number, maximum amount of concurrent requests. Defaults to Infinity. |
{
"mcpServers": {
"website-scraper": {
"command": "npx",
"args": ["-y", "website-scraper"],
"env": {
"DEBUG": "your-value"
}
}
}
}Add to claude_desktop_config.json, then restart Claude Desktop.
| Variable | Description | Required |
|---|---|---|
| DEBUG | Configuration value read at startup. | Optional |
The simplest web tool that matters — fetch any URL and get model-ready markdown back.
Industrial-strength web extraction — render, scrape, crawl and search entire sites into clean markdown.
Puppeteer-powered browser control that drives pages from the accessibility tree instead of pixels.
Search built for AI, not humans — semantic web search that returns model-ready content, plus code context.
Answers, not links — delegate questions to Perplexity's search-grounded models and get cited responses back.
Put 6,000+ pre-built scrapers at your assistant's fingertips through one MCP endpoint.