Crawl4AI
๐ท๏ธ A lightweight Model Context Protocol (MCP) server that exposes Crawl4AI web scraping and crawling capabilities as tools for AI agents. Similar to Firecrawl's API but self-hosted and free. Perfect for integrating web scraping into your AI workflows
What is Crawl4AI?
Crawl4AI is a Model Context Protocol (MCP) server that allows AI assistants like Claude, Cursor, and VS Code to ๐ท๏ธ a lightweight model context protocol (mcp) server that exposes crawl4ai web scraping and crawling capabilities as tools for ai agents. similar to firecrawl's api but self-hosted and free. perfect ...
๐ท๏ธ A lightweight Model Context Protocol (MCP) server that exposes Crawl4AI web scraping and crawling capabilities as tools for AI agents. Similar to Firecrawl's API but self-hosted and free. Perfect for integrating web scraping into your AI workflows
This server falls under the Search & Data Extraction category on MCPgee, the world's largest MCP server directory with 33,000+ servers.
Features
- ๐ท๏ธ A lightweight Model Context Protocol (MCP) server that ex
Use Cases
Maintainer
Works with
Installation
Manual Installation
npx crawl4aiConfiguration
Configuration Details
claude_desktop_config.json
Performance
Response Metrics
Resource Usage
How to Set Up and Use Crawl4AI
Crawl4AI MCP Server is a self-hosted, free alternative to Firecrawl that exposes web scraping and crawling capabilities as MCP tools for AI agents. It wraps the Crawl4AI Python library behind two clean tools โ one for scraping a single page and one for crawling a whole site to a configurable depth โ and returns content as clean markdown optimized for AI consumption. Teams use it to feed current web content to Claude for research, documentation extraction, competitor monitoring, and data pipelines without paying per-page fees or relying on external APIs.
Prerequisites
- Python 3.10 or higher installed
- uv package manager installed (https://astral.sh/uv)
- Git to clone the repository
- An MCP-compatible client such as Claude Desktop, Cursor, or Claude Code
Install the uv package manager
Crawl4AI MCP uses uv for dependency management. Install it with the official installer if you don't have it yet.
curl -LsSf https://astral.sh/uv/install.sh | shClone the repository
Clone the crawl4ai-mcp repository to your local machine.
git clone https://github.com/ritvij14/crawl4ai-mcp.git
cd crawl4ai-mcpCreate a virtual environment and install dependencies
Use uv to create the environment and sync all dependencies from the project's lockfile.
uv venv
uv syncTest the server manually
Activate the virtual environment and run main.py to verify the server starts without errors.
source .venv/bin/activate
python main.pyAdd the server to your MCP client configuration
Configure your MCP client to launch crawl4ai-mcp using uv run. Replace /ABSOLUTE/PATH/TO/PARENT/FOLDER/crawl4ai-mcp with the actual absolute path to the cloned directory.
Crawl4AI Examples
Client configuration
Claude Desktop configuration for the Crawl4AI MCP server using uv run. Update the directory path to match your local clone location.
{
"mcpServers": {
"Crawl4AI": {
"command": "uv",
"args": [
"--directory",
"/absolute/path/to/crawl4ai-mcp",
"run",
"main.py"
]
}
}
}Prompts to try
Example prompts using the scrape_webpage and crawl_website tools.
- "Scrape the content of https://docs.python.org/3/library/asyncio.html and summarize the key concepts"
- "Crawl https://example.com up to 2 levels deep and give me a list of all pages found"
- "Fetch the page at https://news.ycombinator.com and extract the top 10 story titles and links"
- "Crawl https://mysite.com/docs with a depth of 3 and max 10 pages, then find all mentions of 'authentication'"
- "Scrape this product page and extract the price, description, and availability"Troubleshooting Crawl4AI
uv sync fails or dependencies cannot be resolved
Make sure you are inside the cloned crawl4ai-mcp directory before running uv sync. If the issue persists, try deleting the .venv folder and re-running `uv venv && uv sync`.
The server fails to scrape pages behind JavaScript rendering
Crawl4AI uses async browser automation under the hood. Ensure playwright browsers are installed by running `python -m playwright install chromium` inside the activated virtual environment.
MCP client cannot find the server because the directory path is wrong
The --directory argument in the MCP config must be an absolute path. Run `pwd` inside the crawl4ai-mcp directory to get the correct path and paste it into the config.
Frequently Asked Questions about Crawl4AI
What is Crawl4AI?
Crawl4AI is a Model Context Protocol (MCP) server that ๐ท๏ธ a lightweight model context protocol (mcp) server that exposes crawl4ai web scraping and crawling capabilities as tools for ai agents. similar to firecrawl's api but self-hosted and free. perfect for integrating web scraping into your ai workflows It connects AI assistants to external tools and data sources through a standardized interface.
How do I install Crawl4AI?
Follow the installation instructions on the Crawl4AI GitHub repository. Clone the repo, install dependencies, and add the server config to your AI client.
Which AI clients work with Crawl4AI?
Crawl4AI works with all major MCP-compatible AI clients including Claude Desktop, Claude Code, Cursor, VS Code (GitHub Copilot), Windsurf, and Cline.
Is Crawl4AI free to use?
Yes, Crawl4AI is open source and available under the MIT license. You can use it freely in both personal and commercial projects.
Crawl4AI Alternatives โ Similar Search & Data Extraction Servers
Looking for alternatives to Crawl4AI? Here are other popular search & data extraction servers you can use with Claude, Cursor, and VS Code.
TrendRadar
โ 58.0kA real-time hotspot monitoring and news aggregation assistant that provides AI-powered analysis of trending topics across multiple platforms via the Model Context Protocol. It enables users to track news and receive automated notifications through va
Scrapling
โ 52.7k๐ท๏ธ An adaptive Web Scraping framework that handles everything from a single request to a full-scale crawl!
PDF Math Translate
โ 33.9k[EMNLP 2025 Demo] PDF scientific paper translation with preserved formats - ๅบไบ AI ๅฎๆดไฟ็ๆ็็ PDF ๆๆกฃๅ จๆๅ่ฏญ็ฟป่ฏ๏ผๆฏๆ Google/DeepL/Ollama/OpenAI ็ญๆๅก๏ผๆไพ CLI/GUI/MCP/Docker/Zotero
GPT Researcher
โ 27.2kAn autonomous agent that conducts deep research on any data using any LLM providers
Agent Reach
โ 20.1kGive your AI agent eyes to see the entire internet. Read & search Twitter, Reddit, YouTube, GitHub, Bilibili, XiaoHongShu โ one CLI, zero API fees.
Xiaohongshu
โ 13.7kMCP for xiaohongshu.com
Browse More Search & Data Extraction MCP Servers
Explore all search & data extraction servers available in the MCPgee directory. Each server includes setup guides for Claude, Cursor, and VS Code.
Set Up Crawl4AI in Your Editor
Choose your AI client for step-by-step setup instructions.
Quick Config Preview
Add this to your claude_desktop_config.json or .cursor/mcp.json
Ready to use Crawl4AI?
Browse our complete directory of 33,000+ MCP servers, read setup guides for your editor, and start building with the Model Context Protocol.