robotstxt-ai-mcp
Analyze and generate robots.txt files with AI crawler awareness. Fetch any site's robots.txt, detect which AI bots (GPTBot, ClaudeBot, PerplexityBot, Google-Extended) are blocked or allowed, and generate optimized robots.txt with toggle controls for
About
Analyze and generate robots.txt files with AI crawler awareness. Fetch any site's robots.txt, detect which AI bots (GPTBot, ClaudeBot, PerplexityBot, Google-Extended) are blocked or allowed, and generate optimized robots.txt with toggle controls for
Features
- Analyze and generate robots.txt files with AI crawler awaren
Use Cases
Maintainer
Works with
Installation
NPM
npx -y robotstxt-ai-mcpManual Installation
npx -y robotstxt-ai-mcpExamples
No examples available for this server.
Configuration
Configuration Details
claude_desktop_config.json
Not available
Security
Authentication
Features
Performance
Response Metrics
Resource Usage
Similar APIs Servers
Other popular servers in the same category.
n8n-mcp
★ 189.1kA comprehensive MCP server that provides full control over n8n automation workflows through natural language. It offers 43 tools for managing workflows, executions, credentials, and data tables, with safety features like write-mode protection and dou
Dify MCP Server
★ 142.2kProduction-ready platform for agentic workflow development.
gemini-cli-mcp
★ 104.5kA secure MCP server that wraps the Google Gemini CLI, allowing clients to query Gemini models using local OAuth sessions without requiring an API key. It provides tools for model interaction and diagnostics with built-in protection against command in
Awesome Mcp Servers MCP Server
★ 87.3k⭐ Curated list of Model Context Protocol (MCP) servers - tools that extend Claude Desktop, Cursor, Windsurf, and other MCP clients with custom capabilities.
Slack MCP Server
★ 86.1kEnables interaction with Slack workspaces through comprehensive channel management, messaging, user management, file uploads, and Block Kit formatting. Features secure credential storage via macOS Keychain and supports all major Slack operations incl
Servers MCP Server
★ 86.0kModel Context Protocol Servers
Set Up robotstxt-ai-mcp in Your Editor
Choose your AI client for step-by-step setup instructions.
Quick Config Preview
Add this to your claude_desktop_config.json or .cursor/mcp.json
Ready to use robotstxt-ai-mcp?
Browse our complete directory of 33,000+ MCP servers, read setup guides for your editor, and start building with the Model Context Protocol.