Agent Web Scraper
Web scraping MCP server for Al agents. 6 tools: extract clean text/markdown from any URL, structured scraping with CSS selectors, full-page screenshots via Playwright, link extraction with regex filtering, metadata extraction (OG tags, Twitter cards)
What is Agent Web Scraper?
Agent Web Scraper is a Model Context Protocol (MCP) server that allows AI assistants like Claude, Cursor, and VS Code to web scraping mcp server for al agents. 6 tools: extract clean text/markdown from any url, structured scraping with css selectors, full-page screenshots via playwright, link extraction with regex filte...
Web scraping MCP server for Al agents. 6 tools: extract clean text/markdown from any URL, structured scraping with CSS selectors, full-page screenshots via Playwright, link extraction with regex filtering, metadata extraction (OG tags, Twitter cards)
This server falls under the Search & Data Extraction and Browser Automation categories on MCPgee, the world's largest MCP server directory with 33,000+ servers.
Features
- Web scraping MCP server for Al agents. 6 tools: extract clea
Use Cases
Maintainer
Works with
Installation
Manual Installation
npx agent-web-scraperConfiguration
Configuration Details
claude_desktop_config.json
Performance
Response Metrics
Resource Usage
Frequently Asked Questions about Agent Web Scraper
What is Agent Web Scraper?
Agent Web Scraper is a Model Context Protocol (MCP) server that web scraping mcp server for al agents. 6 tools: extract clean text/markdown from any url, structured scraping with css selectors, full-page screenshots via playwright, link extraction with regex filtering, metadata extraction (og tags, twitter cards) It connects AI assistants to external tools and data sources through a standardized interface.
How do I install Agent Web Scraper?
Follow the installation instructions on the Agent Web Scraper GitHub repository. Clone the repo, install dependencies, and add the server config to your AI client.
Which AI clients work with Agent Web Scraper?
Agent Web Scraper works with all major MCP-compatible AI clients including Claude Desktop, Claude Code, Cursor, VS Code (GitHub Copilot), Windsurf, and Cline.
Is Agent Web Scraper free to use?
Yes, Agent Web Scraper is open source and available under the MIT License license. You can use it freely in both personal and commercial projects.
Agent Web Scraper Alternatives — Similar Search & Data Extraction Servers
Looking for alternatives to Agent Web Scraper? Here are other popular search & data extraction servers you can use with Claude, Cursor, and VS Code.
TrendRadar
★ 58.0kA real-time hotspot monitoring and news aggregation assistant that provides AI-powered analysis of trending topics across multiple platforms via the Model Context Protocol. It enables users to track news and receive automated notifications through va
Scrapling
★ 52.7k🕷️ An adaptive Web Scraping framework that handles everything from a single request to a full-scale crawl!
PDF Math Translate
★ 33.9k[EMNLP 2025 Demo] PDF scientific paper translation with preserved formats - 基于 AI 完整保留排版的 PDF 文档全文双语翻译，支持 Google/DeepL/Ollama/OpenAI 等服务，提供 CLI/GUI/MCP/Docker/Zotero
GPT Researcher
★ 27.2kAn autonomous agent that conducts deep research on any data using any LLM providers
Agent Reach
★ 20.1kGive your AI agent eyes to see the entire internet. Read & search Twitter, Reddit, YouTube, GitHub, Bilibili, XiaoHongShu — one CLI, zero API fees.
Xiaohongshu
★ 13.7kMCP for xiaohongshu.com
Browse More Search & Data Extraction MCP Servers
Explore all search & data extraction servers available in the MCPgee directory. Each server includes setup guides for Claude, Cursor, and VS Code.
Set Up Agent Web Scraper in Your Editor
Choose your AI client for step-by-step setup instructions.
Quick Config Preview
Add this to your claude_desktop_config.json or .cursor/mcp.json
Ready to use Agent Web Scraper?
Browse our complete directory of 33,000+ MCP servers, read setup guides for your editor, and start building with the Model Context Protocol.