Documentation Crawler & MCP Server
This project provides a toolset to crawl websites wikis, tool/library documentions and generate Markdown documentation, and make that documentation searchable via a Model Context Protocol (MCP) server, designed for integration with tools like Cursor.
What is Documentation Crawler & MCP Server?
Documentation Crawler & MCP Server is a Model Context Protocol (MCP) server that allows AI assistants like Claude, Cursor, and VS Code to this project provides a toolset to crawl websites wikis, tool/library documentions and generate markdown documentation, and make that documentation searchable via a model context protocol (mcp) server...
This project provides a toolset to crawl websites wikis, tool/library documentions and generate Markdown documentation, and make that documentation searchable via a Model Context Protocol (MCP) server, designed for integration with tools like Cursor.
This server falls under the Search & Data Extraction and Knowledge & Memory categories on MCPgee, the world's largest MCP server directory with 33,000+ servers.
Features
- This project provides a toolset to crawl websites wikis, too
Use Cases
Maintainer
Works with
Installation
Manual Installation
npx documentation-crawler-mcp-serverConfiguration
Configuration Details
claude_desktop_config.json
Performance
Response Metrics
Resource Usage
How to Set Up and Use Documentation Crawler & MCP Server
Documentation Crawler & MCP Server lets you crawl websites, wikis, and library documentation to generate local Markdown files, then makes that content searchable through an MCP interface using vector similarity. It exposes three tools — list_documents, get_document_headings, and search_documentation — that allow AI assistants in Cursor and other MCP clients to semantically query any documentation you have crawled. This is especially useful when working with third-party SDKs or internal wikis that are not indexed in AI training data.
Prerequisites
- Python 3.9+ with uv package manager installed (https://github.com/astral-sh/uv)
- Git to clone the repository
- An MCP-compatible client such as Cursor or Claude Desktop
- No external API keys required — all embedding generation runs locally
Clone the repository
Clone the MCPDocSearch repository to a permanent location on your machine, since the MCP server must reference the absolute path at runtime.
git clone https://github.com/alizdavoodi/MCPDocSearch.git
cd MCPDocSearchInstall dependencies with uv
Run uv sync to create a virtual environment and install all packages declared in pyproject.toml. No manual pip installs are required.
uv syncCrawl a documentation site
Use the crawl.py script to fetch a documentation site and generate local Markdown files. You can limit depth and filter URLs with --max-depth and --include-pattern.
uv run python crawl.py https://docs.example.com --max-depth 2 --include-pattern "*api*"Start the MCP server manually to verify
Run the MCP server directly to confirm it starts without errors. Note that the first startup may take several minutes while it generates embeddings from the crawled content.
uv run python -m mcp_server.mainConfigure Cursor (or Claude Desktop) to use the server
Add the server to your MCP client configuration. For Cursor, create or edit .cursor/mcp.json in your project root. Replace the path with the absolute path where you cloned MCPDocSearch.
Documentation Crawler & MCP Server Examples
Client configuration
Cursor mcp.json configuration pointing to your local MCPDocSearch clone. Replace /absolute/path/to/MCPDocSearch with your actual path.
{
"mcpServers": {
"doc-query-server": {
"command": "uv",
"args": ["--directory", "/absolute/path/to/MCPDocSearch", "run", "python", "-m", "mcp_server.main"],
"env": {}
}
}
}Prompts to try
Example prompts once the server is connected and you have crawled documentation.
- "List all the documents you have indexed"
- "Search the documentation for how to authenticate API requests"
- "Get the heading structure for the quickstart document"
- "Find all references to pagination in the crawled docs"Troubleshooting Documentation Crawler & MCP Server
First server startup is very slow or appears to hang
This is expected — the server generates vector embeddings for all crawled content on first run using a local ML model. On CPU hardware this can take several minutes. Subsequent starts use a cache and are much faster.
search_documentation returns no results
Make sure you have run the crawl.py script first and that it completed successfully. The MCP server can only search content that has been crawled and saved locally.
uv command not found
Install uv with: curl -LsSf https://astral.sh/uv/install.sh | sh — then restart your terminal. Verify with: uv --version
Frequently Asked Questions about Documentation Crawler & MCP Server
What is Documentation Crawler & MCP Server?
Documentation Crawler & MCP Server is a Model Context Protocol (MCP) server that this project provides a toolset to crawl websites wikis, tool/library documentions and generate markdown documentation, and make that documentation searchable via a model context protocol (mcp) server, designed for integration with tools like cursor. It connects AI assistants to external tools and data sources through a standardized interface.
How do I install Documentation Crawler & MCP Server?
Follow the installation instructions on the Documentation Crawler & MCP Server GitHub repository. Clone the repo, install dependencies, and add the server config to your AI client.
Which AI clients work with Documentation Crawler & MCP Server?
Documentation Crawler & MCP Server works with all major MCP-compatible AI clients including Claude Desktop, Claude Code, Cursor, VS Code (GitHub Copilot), Windsurf, and Cline.
Is Documentation Crawler & MCP Server free to use?
Yes, Documentation Crawler & MCP Server is open source and available under the MIT License license. You can use it freely in both personal and commercial projects.
Documentation Crawler & MCP Server Alternatives — Similar Search & Data Extraction Servers
Looking for alternatives to Documentation Crawler & MCP Server? Here are other popular search & data extraction servers you can use with Claude, Cursor, and VS Code.
TrendRadar
★ 58.0kA real-time hotspot monitoring and news aggregation assistant that provides AI-powered analysis of trending topics across multiple platforms via the Model Context Protocol. It enables users to track news and receive automated notifications through va
Scrapling
★ 52.7k🕷️ An adaptive Web Scraping framework that handles everything from a single request to a full-scale crawl!
PDF Math Translate
★ 33.9k[EMNLP 2025 Demo] PDF scientific paper translation with preserved formats - 基于 AI 完整保留排版的 PDF 文档全文双语翻译，支持 Google/DeepL/Ollama/OpenAI 等服务，提供 CLI/GUI/MCP/Docker/Zotero
GPT Researcher
★ 27.2kAn autonomous agent that conducts deep research on any data using any LLM providers
Agent Reach
★ 20.1kGive your AI agent eyes to see the entire internet. Read & search Twitter, Reddit, YouTube, GitHub, Bilibili, XiaoHongShu — one CLI, zero API fees.
Xiaohongshu
★ 13.7kMCP for xiaohongshu.com
Browse More Search & Data Extraction MCP Servers
Explore all search & data extraction servers available in the MCPgee directory. Each server includes setup guides for Claude, Cursor, and VS Code.
Set Up Documentation Crawler & MCP Server in Your Editor
Choose your AI client for step-by-step setup instructions.
Quick Config Preview
Add this to your claude_desktop_config.json or .cursor/mcp.json
Ready to use Documentation Crawler & MCP Server?
Browse our complete directory of 33,000+ MCP servers, read setup guides for your editor, and start building with the Model Context Protocol.