SearxnCrawl
MCP Server and CLI Tools for searxing and fetching websites
What is SearxnCrawl?
SearxnCrawl is a Model Context Protocol (MCP) server that allows AI assistants like Claude, Cursor, and VS Code to mcp server and cli tools for searxing and fetching websites
MCP Server and CLI Tools for searxing and fetching websites
This server falls under the Search & Data Extraction category on MCPgee, the world's largest MCP server directory with 33,000+ servers.
Features
- MCP Server and CLI Tools for searxing and fetching websites
Use Cases
Maintainer
Works with
Installation
Manual Installation
npx searxncrawlConfiguration
Configuration Details
claude_desktop_config.json
Performance
Response Metrics
Resource Usage
How to Set Up and Use SearxnCrawl
SearxnCrawl is a Python MCP server and CLI toolkit that combines a self-hosted SearXNG meta-search engine with a Playwright-powered web crawler, giving AI agents the ability to search the web privately and then fetch and extract full page content in a single workflow. It supports single-page scraping, recursive site crawling with configurable depth and page limits, and content deduplication — producing clean markdown suitable for LLM ingestion without leaking queries to commercial search providers.
Prerequisites
- Python 3.10+ with pip or uv installed
- A running SearXNG instance (can be self-hosted via Docker or a public instance URL)
- Playwright with the Chromium browser: run playwright install chromium after pip install
- Docker and Docker Compose if using the all-in-one setup that bundles SearXNG
- An MCP-compatible client such as Claude Desktop or Cursor
Clone the repository
Clone the searxNcrawl repository to your local machine.
git clone https://github.com/DasDigitaleMomentum/searxNcrawl.git
cd searxNcrawlOption A: All-in-one Docker Compose setup
Copy the example environment file and start both SearXNG and the MCP server together. The MCP server will be available at http://localhost:9555/mcp.
cp .env.example .env
docker compose up --buildOption B: Install with pip for STDIO transport
Create a virtual environment, install dependencies, and install Playwright's Chromium browser for crawling JavaScript-rendered pages.
python -m venv .venv
source .venv/bin/activate
pip install -e .
playwright install chromiumConfigure your MCP client
Add searxNcrawl to your MCP client configuration. Set SEARXNG_URL to point at your SearXNG instance. The example below uses uv to run the server in STDIO mode.
{
"mcpServers": {
"searxncrawl": {
"command": "uv",
"args": ["run", "--directory", "/path/to/searxNcrawl", "python", "-m", "crawler.mcp_server"],
"env": {
"SEARXNG_URL": "http://localhost:8888",
"SEARXNG_USERNAME": "",
"SEARXNG_PASSWORD": ""
}
}
}
}Test with the CLI tools
Before connecting an AI client, verify the tools work using the built-in CLI commands. crawl fetches a single page, search queries SearXNG, and crawl --site performs recursive crawling.
# Single page crawl
crawl https://docs.example.com
# Web search
search "python async tutorial" --max-results 5
# Recursive site crawl
crawl https://docs.example.com --site --max-depth 2 --max-pages 10SearxnCrawl Examples
Client configuration
Claude Desktop configuration for SearxnCrawl using uv in STDIO mode.
{
"mcpServers": {
"searxncrawl": {
"command": "uv",
"args": ["run", "--directory", "/path/to/searxNcrawl", "python", "-m", "crawler.mcp_server"],
"env": {
"SEARXNG_URL": "http://localhost:8888"
}
}
}
}Prompts to try
Example prompts that use SearxnCrawl's search and crawl capabilities.
- "Search for 'Rust async runtimes comparison 2025' and summarize the top 5 results"
- "Crawl https://docs.python.org/3/library/asyncio.html and give me the key concepts"
- "Search for recent news about the EU AI Act and fetch the full text of the top article"
- "Crawl the entire documentation at https://fastapi.tiangolo.com up to depth 2 and list all endpoints covered"
- "Find and fetch three tutorials on PostgreSQL performance tuning"Troubleshooting SearxnCrawl
Search returns no results or connection refused for SearXNG
Verify SEARXNG_URL is set correctly and the SearXNG instance is running. For Docker Compose setups, check docker compose ps. For public SearXNG instances, ensure the URL includes the correct port. Some public instances rate-limit automated queries.
Playwright fails to launch Chromium with 'executable not found'
Run playwright install chromium inside the activated virtual environment to download the browser binary. On Linux servers, you may also need: playwright install-deps chromium.
MCP server port 9555 conflicts with another service
Set MCP_PORT to a different value in your .env file, for example MCP_PORT=9666, and update any client configurations pointing to the old port.
Frequently Asked Questions about SearxnCrawl
What is SearxnCrawl?
SearxnCrawl is a Model Context Protocol (MCP) server that mcp server and cli tools for searxing and fetching websites It connects AI assistants to external tools and data sources through a standardized interface.
How do I install SearxnCrawl?
Follow the installation instructions on the SearxnCrawl GitHub repository. Clone the repo, install dependencies, and add the server config to your AI client.
Which AI clients work with SearxnCrawl?
SearxnCrawl works with all major MCP-compatible AI clients including Claude Desktop, Claude Code, Cursor, VS Code (GitHub Copilot), Windsurf, and Cline.
Is SearxnCrawl free to use?
Yes, SearxnCrawl is open source and available under the MIT license. You can use it freely in both personal and commercial projects.
SearxnCrawl Alternatives — Similar Search & Data Extraction Servers
Looking for alternatives to SearxnCrawl? Here are other popular search & data extraction servers you can use with Claude, Cursor, and VS Code.
TrendRadar
★ 58.0kA real-time hotspot monitoring and news aggregation assistant that provides AI-powered analysis of trending topics across multiple platforms via the Model Context Protocol. It enables users to track news and receive automated notifications through va
Scrapling
★ 52.7k🕷️ An adaptive Web Scraping framework that handles everything from a single request to a full-scale crawl!
PDF Math Translate
★ 33.9k[EMNLP 2025 Demo] PDF scientific paper translation with preserved formats - 基于 AI 完整保留排版的 PDF 文档全文双语翻译，支持 Google/DeepL/Ollama/OpenAI 等服务，提供 CLI/GUI/MCP/Docker/Zotero
GPT Researcher
★ 27.2kAn autonomous agent that conducts deep research on any data using any LLM providers
Agent Reach
★ 20.1kGive your AI agent eyes to see the entire internet. Read & search Twitter, Reddit, YouTube, GitHub, Bilibili, XiaoHongShu — one CLI, zero API fees.
Xiaohongshu
★ 13.7kMCP for xiaohongshu.com
Browse More Search & Data Extraction MCP Servers
Explore all search & data extraction servers available in the MCPgee directory. Each server includes setup guides for Claude, Cursor, and VS Code.
Set Up SearxnCrawl in Your Editor
Choose your AI client for step-by-step setup instructions.
Quick Config Preview
Add this to your claude_desktop_config.json or .cursor/mcp.json
Ready to use SearxnCrawl?
Browse our complete directory of 33,000+ MCP servers, read setup guides for your editor, and start building with the Model Context Protocol.