Analyze and generate robots.txt with AI crawler detection for 20+ bots like GPTBot and ClaudeBot.
Robotstxt-ai is an MCP integration. Analyze and generate robots.txt with AI crawler detection for 20+ bots like GPTBot and ClaudeBot.
The published npm package is robotstxt-ai-mcp, version 1.0.1. Its declared MCP transport is stdio.
Check the maintainer’s instructions for account access, authentication and the supported client configuration. Once connected, inspect the tools your client receives and try a small task that matches the server’s stated purpose. The registry description does not include a verified, complete tool list.
View the publisher’s registry entry · Project repository. This profile uses version 1.0.1 from the registry. Registration identifies the publisher namespace; it does not establish that this is a first-party integration for every service it connects.
Browse related developer tools MCP servers for tools that cover the same kind of work.
Retrieve public web pages as readable Markdown with the reference Fetch MCP server, including pagination for longer responses.
Use Firecrawl MCP to retrieve website content and extract information for research through Firecrawl’s API-backed web tools.
Private context layer for AI agents: web, docs, code search, dep graph, memory, planning.
MCP server to gather usernames from LinkedIn using Serper
35+ web scraping tools. Firecrawl, Bright Data, Jina, Olostep, ScrapeGraph, Notte, Riveter.
A Model Context Protocol server for taking screenshots of web pages using Playwright