Anansi
A self-healing web scraper built for hostile sites: selectors repair themselves, browser rendering kicks in when needed, and Chrome TLS fingerprinting evades bot detection. Ships with an MCP server so any LLM can drive a full crawl through conversati
What is Anansi?
Anansi is a Model Context Protocol (MCP) server that allows AI assistants like Claude, Cursor, and VS Code to self-healing web scraper built for hostile sites: selectors repair themselves, browser rendering kicks in when needed, and chrome tls fingerprinting evades bot detection. ships with an mcp server so a...
A self-healing web scraper built for hostile sites: selectors repair themselves, browser rendering kicks in when needed, and Chrome TLS fingerprinting evades bot detection. Ships with an MCP server so any LLM can drive a full crawl through conversati
This server falls under the Search & Data Extraction category on MCPgee, the world's largest MCP server directory with 33,000+ servers.
Features
- A self-healing web scraper built for hostile sites: selector
Use Cases
Maintainer
Works with
Installation
Manual Installation
npx anansiConfiguration
Configuration Details
claude_desktop_config.json
Performance
Response Metrics
Resource Usage
How to Set Up and Use Anansi
Anansi is a self-healing web scraper MCP server designed to extract structured data from hostile and bot-protected websites. It repairs broken CSS selectors automatically using confidence scoring, upgrades to headless browser rendering when JavaScript is required, and mimics Chrome TLS fingerprints to evade Cloudflare and similar defenses. With 17 exposed MCP tools covering single-page fetching, full-site crawling, selector training, and screenshot capture, any LLM can drive complete data extraction pipelines through conversation.
Prerequisites
- Python 3.10 or later with pip installed
- Playwright and Chromium for browser-based rendering (optional but recommended for JS-heavy sites)
- curl-cffi for TLS fingerprint mimicry (install with the [tls] extra)
- An MCP client such as Claude Desktop or Claude Code
Install Anansi from GitHub
Install the core package. For sites protected by Cloudflare or Akamai, also install the TLS fingerprinting extra.
# Core install
pip install "git+https://github.com/mdowis/anansi"
# With TLS fingerprint mimicry (recommended for bot-protected sites)
pip install "anansi-scraper[tls] @ git+https://github.com/mdowis/anansi"Install Playwright and Chromium for browser rendering
Install Playwright's Chromium binary to enable automatic browser upgrade when JS rendering is detected on a page.
pip install playwright
playwright install chromiumConfigure operator environment variables (optional)
Set environment variables before starting the MCP server to control security and anti-bot behavior. These cannot be changed by the LLM client at runtime.
# Allow scraping internal/private network addresses (off by default)
export ANANSI_ALLOW_PRIVATE_NETWORKS=true
# Set default TLS fingerprint target (e.g. chrome124)
export ANANSI_IMPERSONATE=chrome124
# Disable anti-bot evasion if not needed
# export ANANSI_DISABLE_ANTIBOT=trueRegister the MCP server with your client
Add anansi-mcp as the server command in your Claude Desktop or Claude Code configuration.
{
"mcpServers": {
"anansi": {
"command": "anansi-mcp",
"args": [],
"env": {
"ANANSI_IMPERSONATE": "chrome124"
}
}
}
}For Claude Code, use the CLI shortcut
If you use Claude Code, register the server directly with the mcp add command.
claude mcp add anansi -- anansi-mcpRestart the client and test the connection
Restart your MCP client. Ask it to fetch a webpage or start a crawl to confirm the 17 Anansi tools are available.
Anansi Examples
Client configuration
Claude Desktop JSON config for Anansi with TLS fingerprint impersonation enabled.
{
"mcpServers": {
"anansi": {
"command": "anansi-mcp",
"args": [],
"env": {
"ANANSI_IMPERSONATE": "chrome124"
}
}
}
}Prompts to try
Example prompts that use Anansi's scraping and crawling tools.
- "Fetch https://example.com/products and extract all product names and prices as JSON."
- "Crawl https://news.ycombinator.com and collect the top 30 story titles and URLs."
- "Take a screenshot of https://example.com and show me what it looks like."
- "Start a crawl of https://shop.example.com/category/shoes and export results as CSV when done."
- "Check the selector health for the URL pattern https://example.com/products/*."Troubleshooting Anansi
Scraping fails on Cloudflare-protected sites even with the package installed
Ensure you installed the TLS extra: 'pip install anansi-scraper[tls] @ git+https://github.com/mdowis/anansi'. Then set ANANSI_IMPERSONATE=chrome124 in the server environment. Playwright Chromium must also be installed for sites that serve a JS challenge page.
anansi-mcp command not found after installation
The entry point is installed into your Python environment's bin directory. If using a virtual environment, activate it first. If using pip with --user, ensure ~/.local/bin is in your PATH.
Browser rendering not triggering for JavaScript-heavy pages
Anansi auto-upgrades to browser rendering when it detects SPA markers, but you can force it by passing use_browser=true in the fetch_url tool call. Confirm that 'playwright install chromium' completed without errors.
Frequently Asked Questions about Anansi
What is Anansi?
Anansi is a Model Context Protocol (MCP) server that self-healing web scraper built for hostile sites: selectors repair themselves, browser rendering kicks in when needed, and chrome tls fingerprinting evades bot detection. ships with an mcp server so any llm can drive a full crawl through conversati It connects AI assistants to external tools and data sources through a standardized interface.
How do I install Anansi?
Follow the installation instructions on the Anansi GitHub repository. Clone the repo, install dependencies, and add the server config to your AI client.
Which AI clients work with Anansi?
Anansi works with all major MCP-compatible AI clients including Claude Desktop, Claude Code, Cursor, VS Code (GitHub Copilot), Windsurf, and Cline.
Is Anansi free to use?
Yes, Anansi is open source and available under the Apache-2.0 license. You can use it freely in both personal and commercial projects.
Anansi Alternatives — Similar Search & Data Extraction Servers
Looking for alternatives to Anansi? Here are other popular search & data extraction servers you can use with Claude, Cursor, and VS Code.
TrendRadar
★ 58.0kA real-time hotspot monitoring and news aggregation assistant that provides AI-powered analysis of trending topics across multiple platforms via the Model Context Protocol. It enables users to track news and receive automated notifications through va
Scrapling
★ 52.7k🕷️ An adaptive Web Scraping framework that handles everything from a single request to a full-scale crawl!
PDF Math Translate
★ 33.9k[EMNLP 2025 Demo] PDF scientific paper translation with preserved formats - 基于 AI 完整保留排版的 PDF 文档全文双语翻译，支持 Google/DeepL/Ollama/OpenAI 等服务，提供 CLI/GUI/MCP/Docker/Zotero
GPT Researcher
★ 27.2kAn autonomous agent that conducts deep research on any data using any LLM providers
Agent Reach
★ 20.1kGive your AI agent eyes to see the entire internet. Read & search Twitter, Reddit, YouTube, GitHub, Bilibili, XiaoHongShu — one CLI, zero API fees.
Xiaohongshu
★ 13.7kMCP for xiaohongshu.com
Browse More Search & Data Extraction MCP Servers
Explore all search & data extraction servers available in the MCPgee directory. Each server includes setup guides for Claude, Cursor, and VS Code.
Set Up Anansi in Your Editor
Choose your AI client for step-by-step setup instructions.
Quick Config Preview
Add this to your claude_desktop_config.json or .cursor/mcp.json
Ready to use Anansi?
Browse our complete directory of 33,000+ MCP servers, read setup guides for your editor, and start building with the Model Context Protocol.