Crawl4AI
A locally-hosted MCP server that provides AI assistants with advanced web crawling capabilities, including structured data extraction, deep site crawling, and page screenshots. It enables users to convert single or multiple URLs into clean Markdown c
What is Crawl4AI?
Crawl4AI is a Model Context Protocol (MCP) server that allows AI assistants like Claude, Cursor, and VS Code to locally-hosted mcp server that provides ai assistants with advanced web crawling capabilities, including structured data extraction, deep site crawling, and page screenshots. it enables users to conve...
A locally-hosted MCP server that provides AI assistants with advanced web crawling capabilities, including structured data extraction, deep site crawling, and page screenshots. It enables users to convert single or multiple URLs into clean Markdown c
This server falls under the Search & Data Extraction category on MCPgee, the world's largest MCP server directory with 33,000+ servers.
Features
- A locally-hosted MCP server that provides AI assistants with
Use Cases
Maintainer
Works with
Installation
Manual Installation
npx crawl4ai-mcp-serverConfiguration
Configuration Details
claude_desktop_config.json
Performance
Response Metrics
Resource Usage
How to Set Up and Use Crawl4AI
The Crawl4AI MCP Server is a locally-hosted Python server that gives AI assistants powerful web crawling and content extraction capabilities built on top of the Crawl4AI library and FastMCP. It exposes two primary tools: a multi-engine web search (DuckDuckGo by default, Google with an API key) and a URL reader that converts any webpage into LLM-optimized Markdown with inline citations. The server intelligently filters navigation bars, ads, and other non-content elements while preserving source URLs for attribution. Developers and researchers use it to build AI workflows that require live web research, content summarization, and fact-checking.
Prerequisites
- Python 3.9 or higher installed
- pip and virtualenv for managing Python packages
- Playwright browsers installed via 'playwright install' (required for JavaScript-heavy pages)
- Optional: Google Custom Search API key and CSE ID for Google search support
- An MCP-compatible client such as Claude Desktop
Clone the repository
Download the Crawl4AI MCP Server source code to your local machine.
git clone https://github.com/weidwonder/crawl4ai-mcp-server.git
cd crawl4ai-mcp-serverCreate and activate a virtual environment
Use a dedicated virtual environment to avoid dependency conflicts.
python -m venv crawl4ai_env
source crawl4ai_env/bin/activate # Windows: .\crawl4ai_env\Scripts\activateInstall Python dependencies
Install the packages listed in requirements.txt, which includes crawl4ai, FastMCP, and search clients.
pip install -r requirements.txtInstall Playwright browsers
Crawl4AI uses Playwright for JavaScript rendering. Install the browser binaries with the following command.
playwright installConfigure optional Google search credentials
Copy the demo config and add your Google Custom Search credentials if you want Google search support. DuckDuckGo works with no config.
cp config_demo.json config.json
# Edit config.json and fill in:
# { "google": { "api_key": "YOUR_KEY", "cse_id": "YOUR_CSE_ID" } }Start the MCP server
Run the main server entry point. It communicates via stdio with MCP clients.
python src/index.pyCrawl4AI Examples
Client configuration
Claude Desktop configuration that launches the Crawl4AI MCP server using the virtual environment's Python interpreter.
{
"mcpServers": {
"crawl4ai-mcp-server": {
"command": "/absolute/path/to/crawl4ai-mcp-server/crawl4ai_env/bin/python",
"args": ["/absolute/path/to/crawl4ai-mcp-server/src/index.py"],
"env": {}
}
}
}Prompts to try
Example prompts using the search and read_url tools once the server is running.
- "Search the web for 'Model Context Protocol best practices' and summarize the top 3 results"
- "Fetch the content of https://docs.python.org/3/library/asyncio.html and explain the key concepts"
- "Search for recent blog posts about Rust async programming and show me the titles and URLs"Troubleshooting Crawl4AI
Playwright throws 'Executable doesn't exist' error when crawling
Run 'playwright install' again to download the browser binaries. If behind a corporate proxy, set HTTPS_PROXY before running the install command.
Google search returns an API error or no results
Verify config.json contains valid 'api_key' and 'cse_id' values. The Google Custom Search API must be enabled in your Google Cloud project, and the CSE ID must be configured to search the entire web.
read_url returns mostly empty or garbled content
Try changing the 'format' parameter to 'fit_markdown' which applies aggressive LLM-focused filtering. For sites that heavily rely on JavaScript, ensure Playwright browsers are installed so the crawler can render the page before extracting content.
Frequently Asked Questions about Crawl4AI
What is Crawl4AI?
Crawl4AI is a Model Context Protocol (MCP) server that locally-hosted mcp server that provides ai assistants with advanced web crawling capabilities, including structured data extraction, deep site crawling, and page screenshots. it enables users to convert single or multiple urls into clean markdown c It connects AI assistants to external tools and data sources through a standardized interface.
How do I install Crawl4AI?
Follow the installation instructions on the Crawl4AI GitHub repository. Clone the repo, install dependencies, and add the server config to your AI client.
Which AI clients work with Crawl4AI?
Crawl4AI works with all major MCP-compatible AI clients including Claude Desktop, Claude Code, Cursor, VS Code (GitHub Copilot), Windsurf, and Cline.
Is Crawl4AI free to use?
Yes, Crawl4AI is open source and available under the MIT License license. You can use it freely in both personal and commercial projects.
Crawl4AI Alternatives — Similar Search & Data Extraction Servers
Looking for alternatives to Crawl4AI? Here are other popular search & data extraction servers you can use with Claude, Cursor, and VS Code.
TrendRadar
★ 58.0kA real-time hotspot monitoring and news aggregation assistant that provides AI-powered analysis of trending topics across multiple platforms via the Model Context Protocol. It enables users to track news and receive automated notifications through va
Scrapling
★ 52.7k🕷️ An adaptive Web Scraping framework that handles everything from a single request to a full-scale crawl!
PDF Math Translate
★ 33.9k[EMNLP 2025 Demo] PDF scientific paper translation with preserved formats - 基于 AI 完整保留排版的 PDF 文档全文双语翻译，支持 Google/DeepL/Ollama/OpenAI 等服务，提供 CLI/GUI/MCP/Docker/Zotero
GPT Researcher
★ 27.2kAn autonomous agent that conducts deep research on any data using any LLM providers
Agent Reach
★ 20.1kGive your AI agent eyes to see the entire internet. Read & search Twitter, Reddit, YouTube, GitHub, Bilibili, XiaoHongShu — one CLI, zero API fees.
Xiaohongshu
★ 13.7kMCP for xiaohongshu.com
Browse More Search & Data Extraction MCP Servers
Explore all search & data extraction servers available in the MCPgee directory. Each server includes setup guides for Claude, Cursor, and VS Code.
Set Up Crawl4AI in Your Editor
Choose your AI client for step-by-step setup instructions.
Quick Config Preview
Add this to your claude_desktop_config.json or .cursor/mcp.json
Ready to use Crawl4AI?
Browse our complete directory of 33,000+ MCP servers, read setup guides for your editor, and start building with the Model Context Protocol.