Crawlbase
Crawlbase MCP Server connects AI agents and LLMs with real-time web data. It powers Claude, Cursor, and Windsurf integrations with battle-tested web scraping, JavaScript rendering, and anti-bot protection enabling structured, live data inside your AI
What is Crawlbase?
Crawlbase is a Model Context Protocol (MCP) server that allows AI assistants like Claude, Cursor, and VS Code to crawlbase mcp server connects ai agents and llms with real-time web data. it powers claude, cursor, and windsurf integrations with battle-tested web scraping, javascript rendering, and anti-bot protec...
Crawlbase MCP Server connects AI agents and LLMs with real-time web data. It powers Claude, Cursor, and Windsurf integrations with battle-tested web scraping, JavaScript rendering, and anti-bot protection enabling structured, live data inside your AI
This server falls under the Search & Data Extraction category on MCPgee, the world's largest MCP server directory with 33,000+ servers.
Features
- Crawlbase MCP Server connects AI agents and LLMs with real-t
Use Cases
Maintainer
Works with
Installation
Manual Installation
npx crawlbaseConfiguration
Configuration Details
claude_desktop_config.json
Performance
Response Metrics
Resource Usage
How to Set Up and Use Crawlbase
The Crawlbase MCP Server gives AI agents and LLMs real-time access to live web data by connecting them to the Crawlbase scraping infrastructure, which handles JavaScript rendering, proxy rotation, and anti-bot evasion automatically. It exposes three core tools—fetching raw HTML, extracting clean Markdown, and capturing screenshots—so Claude, Cursor, and Windsurf can retrieve accurate, up-to-date information from any public webpage without hitting bot-detection walls. Development teams and researchers use it when they need structured live data inside AI workflows without building and maintaining their own scraping stack.
Prerequisites
- Node.js 18 or higher with npx available
- A Crawlbase account with a Normal Token (for standard HTML) and a JavaScript Token (for JS-rendered pages) — sign up at crawlbase.com
- An MCP-compatible client such as Claude Desktop, Cursor IDE, or Windsurf
Sign up for Crawlbase and retrieve your API tokens
Create a free Crawlbase account at crawlbase.com. After signup, navigate to your dashboard to copy your Normal Token (for standard HTML pages) and JavaScript Token (for JavaScript-rendered pages and screenshots).
Locate your MCP client configuration file
Find the configuration file for your MCP client. Claude Desktop uses claude_desktop_config.json at ~/Library/Application Support/Claude/ on macOS or %APPDATA%\Claude\ on Windows.
Add Crawlbase to your MCP configuration
Insert the Crawlbase server block into your MCP config, providing both API tokens as environment variables.
{
"mcpServers": {
"crawlbase": {
"type": "stdio",
"command": "npx",
"args": ["@crawlbase/mcp@latest"],
"env": {
"CRAWLBASE_TOKEN": "your_normal_token_here",
"CRAWLBASE_JS_TOKEN": "your_js_token_here"
}
}
}
}Restart your MCP client
Quit and reopen Claude Desktop (or reload your IDE) to load the Crawlbase server. The crawl, crawl_markdown, and crawl_screenshot tools will appear in the available tools list.
Test the integration with a sample crawl
Ask your AI assistant to crawl a webpage. The server will route the request through Crawlbase's infrastructure and return the content.
Crawlbase Examples
Client configuration
Complete claude_desktop_config.json entry for the Crawlbase MCP server with both tokens configured.
{
"mcpServers": {
"crawlbase": {
"type": "stdio",
"command": "npx",
"args": ["@crawlbase/mcp@latest"],
"env": {
"CRAWLBASE_TOKEN": "your_normal_token_here",
"CRAWLBASE_JS_TOKEN": "your_js_token_here"
}
}
}
}Prompts to try
After connecting, use these prompts to extract live web data inside your AI assistant.
- "Crawl Hacker News and return the top 10 stories in markdown format"
- "Fetch the Tesla investor relations page as raw HTML and extract all press release titles"
- "Take a screenshot of the TechCrunch homepage"
- "Get the current pricing page for Vercel as clean markdown"
- "Crawl the GitHub trending page and list the top 5 repositories with their descriptions"Troubleshooting Crawlbase
crawl_screenshot returns an error or blank image
Screenshots require the JavaScript Token (CRAWLBASE_JS_TOKEN), not the Normal Token. Verify that CRAWLBASE_JS_TOKEN is set correctly in the env block and that you have JS crawl credits remaining in your Crawlbase account.
Pages with JavaScript-heavy content return incomplete or empty HTML
Use the crawl_markdown or crawl_screenshot tool (which use CRAWLBASE_JS_TOKEN) instead of crawl. JS-rendered pages require the JavaScript Token to execute scripts before returning content.
Server fails to start with 'npx: command not found'
Ensure Node.js 18+ is installed and npx is on your PATH. Run `npx --version` in a terminal to confirm. On macOS, install Node.js via Homebrew: `brew install node`.
Frequently Asked Questions about Crawlbase
What is Crawlbase?
Crawlbase is a Model Context Protocol (MCP) server that crawlbase mcp server connects ai agents and llms with real-time web data. it powers claude, cursor, and windsurf integrations with battle-tested web scraping, javascript rendering, and anti-bot protection enabling structured, live data inside your ai It connects AI assistants to external tools and data sources through a standardized interface.
How do I install Crawlbase?
Follow the installation instructions on the Crawlbase GitHub repository. Clone the repo, install dependencies, and add the server config to your AI client.
Which AI clients work with Crawlbase?
Crawlbase works with all major MCP-compatible AI clients including Claude Desktop, Claude Code, Cursor, VS Code (GitHub Copilot), Windsurf, and Cline.
Is Crawlbase free to use?
Yes, Crawlbase is open source and available under the MIT license. You can use it freely in both personal and commercial projects.
Crawlbase Alternatives — Similar Search & Data Extraction Servers
Looking for alternatives to Crawlbase? Here are other popular search & data extraction servers you can use with Claude, Cursor, and VS Code.
TrendRadar
★ 58.0kA real-time hotspot monitoring and news aggregation assistant that provides AI-powered analysis of trending topics across multiple platforms via the Model Context Protocol. It enables users to track news and receive automated notifications through va
Scrapling
★ 52.7k🕷️ An adaptive Web Scraping framework that handles everything from a single request to a full-scale crawl!
PDF Math Translate
★ 33.9k[EMNLP 2025 Demo] PDF scientific paper translation with preserved formats - 基于 AI 完整保留排版的 PDF 文档全文双语翻译，支持 Google/DeepL/Ollama/OpenAI 等服务，提供 CLI/GUI/MCP/Docker/Zotero
GPT Researcher
★ 27.2kAn autonomous agent that conducts deep research on any data using any LLM providers
Agent Reach
★ 20.1kGive your AI agent eyes to see the entire internet. Read & search Twitter, Reddit, YouTube, GitHub, Bilibili, XiaoHongShu — one CLI, zero API fees.
Xiaohongshu
★ 13.7kMCP for xiaohongshu.com
Browse More Search & Data Extraction MCP Servers
Explore all search & data extraction servers available in the MCPgee directory. Each server includes setup guides for Claude, Cursor, and VS Code.
Set Up Crawlbase in Your Editor
Choose your AI client for step-by-step setup instructions.
Quick Config Preview
Add this to your claude_desktop_config.json or .cursor/mcp.json
Ready to use Crawlbase?
Browse our complete directory of 33,000+ MCP servers, read setup guides for your editor, and start building with the Model Context Protocol.