Web Scraping
MCP Server leveraging crawl4ai for web scraping and LLM-based content extraction (Markdown, text snippets, smart extraction). Designed for AI agent integration.
What is Web Scraping?
Web Scraping is a Model Context Protocol (MCP) server that allows AI assistants like Claude, Cursor, and VS Code to mcp server leveraging crawl4ai for web scraping and llm-based content extraction (markdown, text snippets, smart extraction). designed for ai agent integration.
MCP Server leveraging crawl4ai for web scraping and LLM-based content extraction (Markdown, text snippets, smart extraction). Designed for AI agent integration.
This server falls under the Search & Data Extraction category on MCPgee, the world's largest MCP server directory with 33,000+ servers.
Features
- MCP Server leveraging crawl4ai for web scraping and LLM-base
Use Cases
Maintainer
Works with
Installation
Manual Installation
npx web-scraping-mcpConfiguration
Configuration Details
claude_desktop_config.json
Performance
Response Metrics
Resource Usage
How to Set Up and Use Web Scraping
The Web Scraping MCP Server uses crawl4ai to fetch and process web pages, exposing three tools to AI assistants: a full-page Markdown scraper, a targeted text snippet extractor, and an LLM-powered smart extractor that uses Google Gemini to return structured data from any URL based on natural language instructions. It is designed for AI agent integration and runs as an SSE server, making it easy to connect to Claude Desktop and other MCP clients over HTTP.
Prerequisites
- Python 3.9 or later with pip installed
- A Google Gemini API key (required for the smart_extract tool)
- Docker (optional, for containerized deployment)
- An MCP client such as Claude Desktop that supports SSE transport
Clone the repository
Clone the web scraping MCP server repository from GitHub.
git clone https://github.com/MaitreyaM/WEB-SCRAPING-MCP.git
cd WEB-SCRAPING-MCPCreate a virtual environment and install dependencies
Set up an isolated Python environment and install the required packages including crawl4ai and FastMCP.
python -m venv venv
source venv/bin/activate # On Windows: venv\Scripts\activate
pip install -r requirements.txtConfigure environment variables
Create a .env file in the project root with your Google Gemini API key. This is required for the smart_extract tool to function.
# .env
GOOGLE_API_KEY=your-google-gemini-api-key-hereStart the MCP server
Run the server script. It starts an SSE server on port 8002 by default. Keep this terminal open while using Claude.
python server.py
# Server listens at http://127.0.0.1:8002/sse
# Or with Docker:
docker build -t crawl4ai-mcp-server .
docker run -it --rm -p 8002:8002 --env-file .env crawl4ai-mcp-serverConfigure your MCP client
Add the web scraping server to your Claude Desktop configuration using the SSE transport URL.
{
"mcpServers": {
"web-scraping-mcp": {
"url": "http://127.0.0.1:8002/sse"
}
}
}Web Scraping Examples
Client configuration
Claude Desktop configuration connecting to the web scraping server via SSE transport on the default port.
{
"mcpServers": {
"web-scraping-mcp": {
"url": "http://127.0.0.1:8002/sse"
}
}
}Prompts to try
These prompts exercise all three tools: full-page Markdown scraping, text snippet extraction, and LLM-powered structured extraction.
- "Scrape the content of https://example.com and give me a summary"
- "Find text snippets about pricing on https://some-saas-product.com"
- "Extract the names, prices, and descriptions of all products listed on https://shop.example.com using smart extraction"
- "Convert the blog post at https://blog.example.com/post to Markdown format"Troubleshooting Web Scraping
smart_extract tool fails with an API key error
Ensure GOOGLE_API_KEY is set in your .env file and the file is in the same directory where you run the server. The smart_extract tool uses Google Gemini and will not function without a valid API key. Get a key at https://aistudio.google.com/.
Claude Desktop cannot connect to the SSE server
Verify the server is running by opening http://127.0.0.1:8002/sse in your browser — you should see an SSE event stream. If the server is not running, start it with `python server.py`. Also confirm the URL in your Claude Desktop config matches exactly (including the /sse path).
crawl4ai fails to fetch JavaScript-heavy pages
crawl4ai supports Playwright for JavaScript rendering. Run `playwright install` after installing requirements to ensure browser drivers are available for pages that require JavaScript execution.
Frequently Asked Questions about Web Scraping
What is Web Scraping?
Web Scraping is a Model Context Protocol (MCP) server that mcp server leveraging crawl4ai for web scraping and llm-based content extraction (markdown, text snippets, smart extraction). designed for ai agent integration. It connects AI assistants to external tools and data sources through a standardized interface.
How do I install Web Scraping?
Follow the installation instructions on the Web Scraping GitHub repository. Clone the repo, install dependencies, and add the server config to your AI client.
Which AI clients work with Web Scraping?
Web Scraping works with all major MCP-compatible AI clients including Claude Desktop, Claude Code, Cursor, VS Code (GitHub Copilot), Windsurf, and Cline.
Is Web Scraping free to use?
Yes, Web Scraping is open source and available under the MIT license. You can use it freely in both personal and commercial projects.
Web Scraping Alternatives — Similar Search & Data Extraction Servers
Looking for alternatives to Web Scraping? Here are other popular search & data extraction servers you can use with Claude, Cursor, and VS Code.
TrendRadar
★ 58.0kA real-time hotspot monitoring and news aggregation assistant that provides AI-powered analysis of trending topics across multiple platforms via the Model Context Protocol. It enables users to track news and receive automated notifications through va
Scrapling
★ 52.7k🕷️ An adaptive Web Scraping framework that handles everything from a single request to a full-scale crawl!
PDF Math Translate
★ 33.9k[EMNLP 2025 Demo] PDF scientific paper translation with preserved formats - 基于 AI 完整保留排版的 PDF 文档全文双语翻译，支持 Google/DeepL/Ollama/OpenAI 等服务，提供 CLI/GUI/MCP/Docker/Zotero
GPT Researcher
★ 27.2kAn autonomous agent that conducts deep research on any data using any LLM providers
Agent Reach
★ 20.1kGive your AI agent eyes to see the entire internet. Read & search Twitter, Reddit, YouTube, GitHub, Bilibili, XiaoHongShu — one CLI, zero API fees.
Xiaohongshu
★ 13.7kMCP for xiaohongshu.com
Browse More Search & Data Extraction MCP Servers
Explore all search & data extraction servers available in the MCPgee directory. Each server includes setup guides for Claude, Cursor, and VS Code.
Set Up Web Scraping in Your Editor
Choose your AI client for step-by-step setup instructions.
Quick Config Preview
Add this to your claude_desktop_config.json or .cursor/mcp.json
Ready to use Web Scraping?
Browse our complete directory of 33,000+ MCP servers, read setup guides for your editor, and start building with the Model Context Protocol.