MCPToolBench++
MCPToolBench++ MCP Model Context Protocol Tool Use Benchmark on AI Agent and Model Tool Use Ability
What is MCPToolBench++?
MCPToolBench++ is a Model Context Protocol (MCP) server that allows AI assistants like Claude, Cursor, and VS Code to mcptoolbench++ mcp model context protocol tool use benchmark on ai agent and model tool use ability
MCPToolBench++ MCP Model Context Protocol Tool Use Benchmark on AI Agent and Model Tool Use Ability
This server falls under the Developer Tools category on MCPgee, the world's largest MCP server directory with 33,000+ servers.
Features
- MCPToolBench++ MCP Model Context Protocol Tool Use Benchmark
Use Cases
Maintainer
Works with
Installation
Manual Installation
npx mcptoolbenchppConfiguration
Configuration Details
claude_desktop_config.json
Performance
Response Metrics
Resource Usage
How to Set Up and Use MCPToolBench++
MCPToolBench++ is a research benchmark framework for evaluating how well AI models and agents use MCP-exposed tools across categories including browser automation, file system, web search, maps, finance, and payment. It ships with a marketplace of real MCP servers, an evaluation harness that measures pass@k rates, and an LLM-as-judge scoring system. AI researchers and model developers use it to compare tool-use capabilities across models such as GPT-4o, Claude, Qwen, and Mistral on standardized task suites.
Prerequisites
- Python 3.9+ with pip installed
- API keys for the models you want to benchmark (OPENAI_API_KEY, ANTHROPIC_API_KEY, QWEN_API_KEY, etc.)
- API keys for tool categories under test (TAVILY_API_KEY, GOOGLE_MAPS_API_KEY, etc.)
- Git for cloning the repository and its submodules
- An MCP-compatible runtime for running the marketplace servers
Install via PyPI (recommended)
Install the mcp-marketplace package and the optional MCP tool use evaluation extras.
pip install mcp-marketplace
pip install mcp-marketplace[mcp_tool_use]Clone the full repository for benchmark tasks
For access to the evaluation task data files and the full harness, clone the MCPToolBenchPP repo and its mcp-marketplace submodule.
git clone https://github.com/mcp-tool-bench/MCPToolBenchPP
cd MCPToolBenchPP/mcp
git clone https://github.com/aiagenta2z/mcp-marketplaceSet required environment variables
Export API keys for each model and tool category you plan to test. At minimum, set keys for the model under evaluation and the search or maps tools used in your benchmark category.
export ANTHROPIC_API_KEY=your_anthropic_key
export OPENAI_API_KEY=your_openai_key
export QWEN_API_KEY=your_qwen_key
export TAVILY_API_KEY=your_tavily_key
export GOOGLE_MAPS_API_KEY=your_maps_keyStart the MCP marketplace server
Launch the marketplace server that provides the tool endpoints the benchmark agents will call. Use --config to point to a custom tool configuration file.
mcpm run --host 0.0.0.0 --port 5000 --config ./mcp_config.jsonRun a benchmark evaluation
Execute the benchmark harness against a specific category and model. The --pass_k flag controls how many attempts per task, and --llm_as_judge_model sets which model scores the outputs.
python3 run.py \
--stage tool_call \
--input_file ./data/browser/browser_demo.json \
--category browser \
--model qwen3-coder-plus \
--pass_k 1 \
--evaluation_trial_per_task 5 \
--llm_as_judge_model qwen-plusMCPToolBench++ Examples
Client configuration
claude_desktop_config.json entry for connecting an AI client to the MCPToolBench++ marketplace server running locally on port 5000.
{
"mcpServers": {
"mcptoolbench": {
"command": "mcpm",
"args": ["run", "--port", "5000", "--config", "/path/to/mcp_config.json"]
}
}
}Prompts to try
Example research and evaluation prompts when using the benchmark tools interactively through an MCP client.
- "Run the browser benchmark category against Claude and report pass@1 rates."
- "Compare GPT-4o and Qwen tool use performance on the search category tasks."
- "Use tavily-search to find recent papers on MCP tool use benchmarking."
- "Run maps_direction_driving from Times Square to JFK and report estimated travel time."
- "Execute 5 trials of the file system task suite and compute average LLM-judge score."Troubleshooting MCPToolBench++
mcpm: command not found after pip install
Ensure the pip scripts directory is on your PATH. Run python3 -m pip show mcp-marketplace to find the install location, then add the bin directory to PATH. On macOS/Linux: export PATH="$HOME/.local/bin:$PATH".
Benchmark run fails with missing API key errors for tool categories
Each tool category requires its own API key. Check which category you are running (browser, search, maps, finance, payment) and export the corresponding key. See the README for the full mapping of categories to required environment variables.
LLM-as-judge scores are inconsistent across runs
This is expected behaviour due to LLM non-determinism. Use a higher --evaluation_trial_per_task value (5–10) and report averaged pass@k scores. The --pass_k flag controls the k in pass@k, not the number of trials.
Frequently Asked Questions about MCPToolBench++
What is MCPToolBench++?
MCPToolBench++ is a Model Context Protocol (MCP) server that mcptoolbench++ mcp model context protocol tool use benchmark on ai agent and model tool use ability It connects AI assistants to external tools and data sources through a standardized interface.
How do I install MCPToolBench++?
Follow the installation instructions on the MCPToolBench++ GitHub repository. Clone the repo, install dependencies, and add the server config to your AI client.
Which AI clients work with MCPToolBench++?
MCPToolBench++ works with all major MCP-compatible AI clients including Claude Desktop, Claude Code, Cursor, VS Code (GitHub Copilot), Windsurf, and Cline.
Is MCPToolBench++ free to use?
Yes, MCPToolBench++ is open source and available under the MIT license. You can use it freely in both personal and commercial projects.
MCPToolBench++ Alternatives — Similar Developer Tools Servers
Looking for alternatives to MCPToolBench++? Here are other popular developer tools servers you can use with Claude, Cursor, and VS Code.
Ecc
★ 188.2kThe agent harness performance optimization system. Skills, instincts, memory, security, and research-first development for Claude Code, Codex, Opencode, Cursor and beyond.
Javaguide
★ 155.8kJava 面试 & 后端通用面试指南，覆盖计算机基础、数据库、分布式、高并发、系统设计与 AI 应用开发
Gemini CLI
★ 104.5kA secure MCP server that wraps the Google Gemini CLI, allowing clients to query Gemini models using local OAuth sessions without requiring an API key. It provides tools for model interaction and diagnostics with built-in protection against command in
Awesome MCP Servers
★ 87.3k⭐ Curated list of Model Context Protocol (MCP) servers - tools that extend Claude Desktop, Cursor, Windsurf, and other MCP clients with custom capabilities.
MCP Servers
★ 86.0kModel Context Protocol Servers
CC Switch
★ 77.5kA cross-platform desktop All-in-One assistant for Claude Code, Codex, OpenCode, OpenClaw, Gemini CLI & Hermes Agent. Only official website: ccswitch.io
Browse More Developer Tools MCP Servers
Explore all developer tools servers available in the MCPgee directory. Each server includes setup guides for Claude, Cursor, and VS Code.
Set Up MCPToolBench++ in Your Editor
Choose your AI client for step-by-step setup instructions.
Quick Config Preview
Add this to your claude_desktop_config.json or .cursor/mcp.json
Ready to use MCPToolBench++?
Browse our complete directory of 33,000+ MCP servers, read setup guides for your editor, and start building with the Model Context Protocol.