Crawl MCP Server
An MCP server for web content extraction that converts HTML pages into clean, LLM-optimized Markdown using Mozilla's Readability. It supports batch processing, intelligent multi-page crawling, and configurable caching while respecting robots.txt stan
What is Crawl MCP Server?
Crawl MCP Server is a Model Context Protocol (MCP) server that allows AI assistants like Claude, Cursor, and VS Code to mcp server for web content extraction that converts html pages into clean, llm-optimized markdown using mozilla's readability. it supports batch processing, intelligent multi-page crawling, and config...
An MCP server for web content extraction that converts HTML pages into clean, LLM-optimized Markdown using Mozilla's Readability. It supports batch processing, intelligent multi-page crawling, and configurable caching while respecting robots.txt stan
This server falls under the Search & Data Extraction category on MCPgee, the world's largest MCP server directory with 33,000+ servers.
Features
- An MCP server for web content extraction that converts HTML
Use Cases
Maintainer
Works with
Installation
Manual Installation
npx crawlConfiguration
Configuration Details
claude_desktop_config.json
Performance
Response Metrics
Resource Usage
How to Set Up and Use Crawl MCP Server
The Crawl MCP Server is a web content extraction server that fetches HTML pages and converts them to clean, LLM-optimized Markdown using Mozilla's Readability library, the same engine that powers Firefox Reader Mode. It supports single-page extraction, batch processing of URL lists, and intelligent multi-page crawling that follows links depth-first while respecting robots.txt directives. Fetched pages are cached to avoid redundant network requests, and the output strips navigation, ads, and boilerplate so that only the main article content is passed to the model. Developers use it to feed real-time web content into AI workflows without building their own scraping pipeline.
Prerequisites
- Node.js 18 or newer installed
- npm or npx available on the command line
- Internet access from the machine running the server
- An MCP-compatible client such as Claude Desktop or Cursor
Clone the repository
Clone the searchcrawl-mcp-server repository from GitHub and enter the project directory.
git clone https://github.com/Git-Fg/searchcrawl-mcp-server.git
cd searchcrawl-mcp-serverInstall dependencies
Install all Node.js dependencies including the Readability parser and MCP SDK.
npm installBuild the TypeScript source
Compile the TypeScript source files to JavaScript so they can be run with Node.
npm run buildTest the server locally
Start the server manually to confirm it runs without errors before connecting a client.
node dist/index.jsAdd to Claude Desktop configuration
Add the crawl server to your Claude Desktop config file so it launches automatically when Claude starts.
{
"mcpServers": {
"crawl": {
"command": "node",
"args": ["/absolute/path/to/searchcrawl-mcp-server/dist/index.js"]
}
}
}Crawl MCP Server Examples
Client configuration (Claude Desktop)
Configuration block for Claude Desktop pointing to the locally built crawl MCP server.
{
"mcpServers": {
"crawl": {
"command": "node",
"args": ["/absolute/path/to/searchcrawl-mcp-server/dist/index.js"]
}
}
}Prompts to try
Example prompts that use the crawl server to extract and summarize web content.
- "Fetch the content of https://example.com/article and summarize the main points."
- "Crawl https://docs.example.com starting from the index page, follow up to 3 levels of links, and extract all content."
- "Fetch these three URLs and compare their content: [url1, url2, url3]."
- "Extract the main article text from https://news.example.com/story and translate it to Spanish."Troubleshooting Crawl MCP Server
Fetched pages return empty content or only boilerplate text.
Some pages require JavaScript rendering and cannot be parsed with the static Readability approach. Try a different URL or a page that serves full HTML server-side. JavaScript-heavy SPAs may need a headless browser tool instead.
Crawling is blocked by robots.txt on certain sites.
The server respects robots.txt by design. If you own the target site, ensure your robots.txt allows the crawler. For sites that block automated access, you will need to fetch individual pages manually.
node dist/index.js fails with 'Cannot find module' errors.
Run 'npm install' followed by 'npm run build' again to ensure all dependencies are installed and compiled. Check that you are running the command from the project root directory.
Frequently Asked Questions about Crawl MCP Server
What is Crawl MCP Server?
Crawl MCP Server is a Model Context Protocol (MCP) server that mcp server for web content extraction that converts html pages into clean, llm-optimized markdown using mozilla's readability. it supports batch processing, intelligent multi-page crawling, and configurable caching while respecting robots.txt stan It connects AI assistants to external tools and data sources through a standardized interface.
How do I install Crawl MCP Server?
Follow the installation instructions on the Crawl MCP Server GitHub repository. Clone the repo, install dependencies, and add the server config to your AI client.
Which AI clients work with Crawl MCP Server?
Crawl MCP Server works with all major MCP-compatible AI clients including Claude Desktop, Claude Code, Cursor, VS Code (GitHub Copilot), Windsurf, and Cline.
Is Crawl MCP Server free to use?
Yes, Crawl MCP Server is open source and available under the MIT license. You can use it freely in both personal and commercial projects.
Crawl MCP Server Alternatives — Similar Search & Data Extraction Servers
Looking for alternatives to Crawl MCP Server? Here are other popular search & data extraction servers you can use with Claude, Cursor, and VS Code.
TrendRadar
★ 58.0kA real-time hotspot monitoring and news aggregation assistant that provides AI-powered analysis of trending topics across multiple platforms via the Model Context Protocol. It enables users to track news and receive automated notifications through va
Scrapling
★ 52.7k🕷️ An adaptive Web Scraping framework that handles everything from a single request to a full-scale crawl!
PDF Math Translate
★ 33.9k[EMNLP 2025 Demo] PDF scientific paper translation with preserved formats - 基于 AI 完整保留排版的 PDF 文档全文双语翻译，支持 Google/DeepL/Ollama/OpenAI 等服务，提供 CLI/GUI/MCP/Docker/Zotero
GPT Researcher
★ 27.2kAn autonomous agent that conducts deep research on any data using any LLM providers
Agent Reach
★ 20.1kGive your AI agent eyes to see the entire internet. Read & search Twitter, Reddit, YouTube, GitHub, Bilibili, XiaoHongShu — one CLI, zero API fees.
Xiaohongshu
★ 13.7kMCP for xiaohongshu.com
Browse More Search & Data Extraction MCP Servers
Explore all search & data extraction servers available in the MCPgee directory. Each server includes setup guides for Claude, Cursor, and VS Code.
Set Up Crawl MCP Server in Your Editor
Choose your AI client for step-by-step setup instructions.
Quick Config Preview
Add this to your claude_desktop_config.json or .cursor/mcp.json
Ready to use Crawl MCP Server?
Browse our complete directory of 33,000+ MCP servers, read setup guides for your editor, and start building with the Model Context Protocol.