WebScraping.AI
A Model Context Protocol (MCP) server implementation that integrates with WebScraping.AI for web data extraction capabilities.
What is WebScraping.AI?
WebScraping.AI is a Model Context Protocol (MCP) server that allows AI assistants like Claude, Cursor, and VS Code to model context protocol (mcp) server implementation that integrates with webscraping.ai for web data extraction capabilities.
A Model Context Protocol (MCP) server implementation that integrates with WebScraping.AI for web data extraction capabilities.
This server falls under the Search & Data Extraction category on MCPgee, the world's largest MCP server directory with 33,000+ servers.
Features
- A Model Context Protocol (MCP) server implementation that in
Use Cases
Maintainer
Works with
Installation
Manual Installation
npx webscraping-ai-mcp-serverConfiguration
Configuration Details
claude_desktop_config.json
Performance
Response Metrics
Resource Usage
How to Set Up and Use WebScraping.AI
The WebScraping.AI MCP Server integrates the WebScraping.AI cloud API into any MCP-compatible AI client, giving it the ability to extract HTML, text, and structured data from websites that use JavaScript rendering, dynamic content, and bot-detection systems. It exposes seven tools covering full HTML retrieval, visible text extraction, CSS selector-based element picking, field extraction, and AI-powered question answering about page content — all using residential proxies by default. Developers and data teams use it to build AI-driven web data pipelines without managing browser automation infrastructure themselves.
Prerequisites
- A WebScraping.AI account and API key — sign up at webscraping.ai to obtain WEBSCRAPING_AI_API_KEY
- Node.js 18+ for npx execution
- An MCP-compatible client such as Claude Desktop, Cursor, or Claude Code
Obtain a WebScraping.AI API key
Sign up at webscraping.ai and copy your API key from the dashboard. This becomes the required WEBSCRAPING_AI_API_KEY environment variable.
Test the server quickly with npx
Run the server directly with npx to confirm your API key works before adding it to your permanent MCP config.
env WEBSCRAPING_AI_API_KEY=your_api_key npx -y webscraping-ai-mcpAdd the server to your MCP client configuration
Edit claude_desktop_config.json to register the WebScraping.AI server with your API key and any optional configuration parameters.
{
"mcpServers": {
"webscraping-ai": {
"command": "npx",
"args": ["-y", "webscraping-ai-mcp"],
"env": {
"WEBSCRAPING_AI_API_KEY": "your-api-key",
"WEBSCRAPING_AI_DEFAULT_PROXY_TYPE": "residential",
"WEBSCRAPING_AI_DEFAULT_JS_RENDERING": "true"
}
}
}
}Restart your MCP client
Restart Claude Desktop or your chosen client to load the new server. Seven WebScraping.AI tools should appear in the tool panel.
Enable content sandboxing for untrusted pages (optional)
If scraping untrusted sites, set WEBSCRAPING_AI_ENABLE_CONTENT_SANDBOXING to true. This wraps scraped content with security boundaries to protect against prompt injection attacks.
"WEBSCRAPING_AI_ENABLE_CONTENT_SANDBOXING": "true"WebScraping.AI Examples
Client configuration
Complete claude_desktop_config.json entry with the required API key and common optional settings.
{
"mcpServers": {
"webscraping-ai": {
"command": "npx",
"args": ["-y", "webscraping-ai-mcp"],
"env": {
"WEBSCRAPING_AI_API_KEY": "your-api-key-here",
"WEBSCRAPING_AI_DEFAULT_PROXY_TYPE": "residential",
"WEBSCRAPING_AI_DEFAULT_JS_RENDERING": "true",
"WEBSCRAPING_AI_CONCURRENCY_LIMIT": "5",
"WEBSCRAPING_AI_ENABLE_CONTENT_SANDBOXING": "false"
}
}
}
}Prompts to try
Example prompts covering the seven WebScraping.AI MCP tools.
- "Extract the full HTML of https://example.com/products with JavaScript rendering enabled"
- "Get the visible text content from https://news.ycombinator.com"
- "Extract the product name, price, and description fields from https://example.com/product/123"
- "Ask: what is the return policy on https://example.com/faq?"
- "Show me my WebScraping.AI account usage and remaining API credits"Troubleshooting WebScraping.AI
401 Unauthorized — API key invalid
Verify WEBSCRAPING_AI_API_KEY is set to the exact key from your webscraping.ai dashboard. Keys are case-sensitive. Try the npx quickstart command in a terminal first to isolate the issue.
Scraping times out on JavaScript-heavy pages
Increase WEBSCRAPING_AI_DEFAULT_TIMEOUT (max 30000ms) and WEBSCRAPING_AI_DEFAULT_JS_TIMEOUT in your env config. Some pages require additional wait time for dynamic content to load.
Scraped content contains bot-detection blocks or CAPTCHAs
Switch WEBSCRAPING_AI_DEFAULT_PROXY_TYPE to 'stealth' for sites with aggressive bot detection. The stealth proxy type uses more sophisticated evasion techniques at a higher API cost.
Frequently Asked Questions about WebScraping.AI
What is WebScraping.AI?
WebScraping.AI is a Model Context Protocol (MCP) server that model context protocol (mcp) server implementation that integrates with webscraping.ai for web data extraction capabilities. It connects AI assistants to external tools and data sources through a standardized interface.
How do I install WebScraping.AI?
Follow the installation instructions on the WebScraping.AI GitHub repository. Clone the repo, install dependencies, and add the server config to your AI client.
Which AI clients work with WebScraping.AI?
WebScraping.AI works with all major MCP-compatible AI clients including Claude Desktop, Claude Code, Cursor, VS Code (GitHub Copilot), Windsurf, and Cline.
Is WebScraping.AI free to use?
Yes, WebScraping.AI is open source and available under the MIT License license. You can use it freely in both personal and commercial projects.
WebScraping.AI Alternatives — Similar Search & Data Extraction Servers
Looking for alternatives to WebScraping.AI? Here are other popular search & data extraction servers you can use with Claude, Cursor, and VS Code.
TrendRadar
★ 58.0kA real-time hotspot monitoring and news aggregation assistant that provides AI-powered analysis of trending topics across multiple platforms via the Model Context Protocol. It enables users to track news and receive automated notifications through va
Scrapling
★ 52.7k🕷️ An adaptive Web Scraping framework that handles everything from a single request to a full-scale crawl!
PDF Math Translate
★ 33.9k[EMNLP 2025 Demo] PDF scientific paper translation with preserved formats - 基于 AI 完整保留排版的 PDF 文档全文双语翻译，支持 Google/DeepL/Ollama/OpenAI 等服务，提供 CLI/GUI/MCP/Docker/Zotero
GPT Researcher
★ 27.2kAn autonomous agent that conducts deep research on any data using any LLM providers
Agent Reach
★ 20.1kGive your AI agent eyes to see the entire internet. Read & search Twitter, Reddit, YouTube, GitHub, Bilibili, XiaoHongShu — one CLI, zero API fees.
Xiaohongshu
★ 13.7kMCP for xiaohongshu.com
Browse More Search & Data Extraction MCP Servers
Explore all search & data extraction servers available in the MCPgee directory. Each server includes setup guides for Claude, Cursor, and VS Code.
Set Up WebScraping.AI in Your Editor
Choose your AI client for step-by-step setup instructions.
Quick Config Preview
Add this to your claude_desktop_config.json or .cursor/mcp.json
Ready to use WebScraping.AI?
Browse our complete directory of 33,000+ MCP servers, read setup guides for your editor, and start building with the Model Context Protocol.