Tokio Prompt Orchestrator
Multi-core, Tokio-native orchestration for LLM pipelines.
What is Tokio Prompt Orchestrator?
Tokio Prompt Orchestrator is a Model Context Protocol (MCP) server that allows AI assistants like Claude, Cursor, and VS Code to multi-core, tokio-native orchestration for llm pipelines.
Multi-core, Tokio-native orchestration for LLM pipelines.
This server falls under the Coding Agents category on MCPgee, the world's largest MCP server directory with 33,000+ servers.
Features
- Multi-core, Tokio-native orchestration for LLM pipelines.
Use Cases
Maintainer
Works with
Installation
Manual Installation
npx tokio-prompt-orchestratorConfiguration
Configuration Details
claude_desktop_config.json
Performance
Response Metrics
Resource Usage
How to Set Up and Use Tokio Prompt Orchestrator
Tokio Prompt Orchestrator is a Rust-native, async-first MCP server that turns any LLM provider into a resilient, observable multi-stage pipeline. Built on the Tokio async runtime, it supports Anthropic, OpenAI, llama.cpp, and vLLM backends simultaneously and exposes circuit breakers, semantic deduplication, token-budget enforcement, and Kalman-filter-tuned adaptive worker pools. Developers building production AI workloads use it to add retry logic, A/B test providers, stream results over SSE or WebSocket, and capture full audit logs — all through a single MCP interface that plugs into Claude Desktop or any MCP-compatible client.
Prerequisites
- Rust toolchain (stable, 1.75+) and Cargo installed for source builds
- ANTHROPIC_API_KEY or OPENAI_API_KEY for cloud LLM backends (or a running llama.cpp/vLLM local server)
- An MCP-compatible client such as Claude Desktop
- Optional: Docker and Redis for distributed/caching deployments
- Optional: Jaeger or compatible OpenTelemetry collector for tracing
Clone the repository and build the MCP binary
Clone the repo and compile with the mcp feature flag enabled. The mcp feature pulls in the MCP protocol layer on top of the core orchestration engine.
git clone https://github.com/Mattbusel/tokio-prompt-orchestrator
cd tokio-prompt-orchestrator
cargo build --release --bin mcp --features mcpSet required environment variables
Export the API key for your chosen backend. At minimum one of ANTHROPIC_API_KEY or OPENAI_API_KEY is needed. Set RUST_LOG to control log verbosity.
export ANTHROPIC_API_KEY=sk-ant-...
export RUST_LOG=infoCreate a pipeline configuration file
Write a pipeline.toml that defines your inference stage, resilience settings, and rate limits. The worker field selects the backend: anthropic, open_ai, llama_cpp, vllm, or echo (for testing).
[pipeline]
name = "production"
[stages.inference]
worker = "anthropic"
model = "claude-sonnet-4-6"
max_tokens = 1024
[resilience]
retry_attempts = 3
circuit_breaker_threshold = 5
circuit_breaker_timeout_s = 60
[rate_limits]
enabled = true
requests_per_second = 100Start the MCP server
Run the compiled mcp binary. It listens on localhost:8080 by default and exposes the MCP protocol for client connections.
./target/release/mcp --config pipeline.tomlAdd to Claude Desktop configuration
Open claude_desktop_config.json and add the orchestrator entry pointing to the running HTTP server. Use the url field to connect over Streamable HTTP.
{
"mcpServers": {
"orchestrator": {
"url": "http://127.0.0.1:8080"
}
}
}Tokio Prompt Orchestrator Examples
Client configuration
Claude Desktop configuration connecting to a locally running Tokio Prompt Orchestrator MCP server over HTTP.
{
"mcpServers": {
"orchestrator": {
"url": "http://127.0.0.1:8080"
}
}
}Prompts to try
Example prompts that leverage the orchestrator's pipeline management and inference tools.
- "Send the prompt 'Summarize the Rust ownership model' through the orchestrator pipeline and return the result."
- "Check the pipeline status — are there any circuit breakers open or requests in the dead-letter queue?"
- "Run a batch inference on these 5 prompts and tell me which ones failed."
- "Replay the failed request with ID abc123 from the dead-letter queue."
- "Update the pipeline configuration to use the open_ai worker instead of anthropic."Troubleshooting Tokio Prompt Orchestrator
cargo build fails with missing features error
Ensure you pass --features mcp (or --features full for the full production bundle) to cargo build. Without explicit feature flags the MCP transport code is excluded from the build.
Circuit breaker trips immediately on startup
Check that your API key is valid and that the model name in pipeline.toml matches an available model. Verify with RUST_LOG=debug to see the raw error from the provider before the circuit breaker engages.
Claude Desktop cannot connect to the orchestrator
Confirm the MCP binary is running and listening on port 8080 (check with lsof -i :8080). The claude_desktop_config.json must use the url field (not command/args) because the orchestrator runs as a standalone HTTP server, not a subprocess.
Frequently Asked Questions about Tokio Prompt Orchestrator
What is Tokio Prompt Orchestrator?
Tokio Prompt Orchestrator is a Model Context Protocol (MCP) server that multi-core, tokio-native orchestration for llm pipelines. It connects AI assistants to external tools and data sources through a standardized interface.
How do I install Tokio Prompt Orchestrator?
Follow the installation instructions on the Tokio Prompt Orchestrator GitHub repository. Clone the repo, install dependencies, and add the server config to your AI client.
Which AI clients work with Tokio Prompt Orchestrator?
Tokio Prompt Orchestrator works with all major MCP-compatible AI clients including Claude Desktop, Claude Code, Cursor, VS Code (GitHub Copilot), Windsurf, and Cline.
Is Tokio Prompt Orchestrator free to use?
Yes, Tokio Prompt Orchestrator is open source and available under the MIT license. You can use it freely in both personal and commercial projects.
Tokio Prompt Orchestrator Alternatives — Similar Coding Agents Servers
Looking for alternatives to Tokio Prompt Orchestrator? Here are other popular coding agents servers you can use with Claude, Cursor, and VS Code.
Dify
★ 142.2kProduction-ready platform for agentic workflow development.
Ruflo
★ 54.0k🌊 The leading agent orchestration platform for Claude. Deploy intelligent multi-agent swarms, coordinate autonomous workflows, and build conversational AI systems. Features enterprise-grade architecture, self-learning swarm intelligence, RAG integrat
Goose
★ 45.7kan open source, extensible AI agent that goes beyond code suggestions - install, execute, edit, and test with any LLM
Antigravity Awesome Skills
★ 38.3kInstallable GitHub library of 1,400+ agentic skills for Claude Code, Cursor, Codex CLI, Gemini CLI, Antigravity, and more. Includes installer CLI, bundles, workflows, and official/community skill collections.
AgentScope
★ 25.5kBuild and run agents you can see, understand and trust.
Serena
★ 24.5kA coding agent toolkit that provides IDE-like semantic code retrieval and editing tools, enabling LLMs to efficiently navigate and modify codebases using symbol-level operations instead of basic file reading and string replacements.
Browse More Coding Agents MCP Servers
Explore all coding agents servers available in the MCPgee directory. Each server includes setup guides for Claude, Cursor, and VS Code.
Set Up Tokio Prompt Orchestrator in Your Editor
Choose your AI client for step-by-step setup instructions.
Quick Config Preview
Add this to your claude_desktop_config.json or .cursor/mcp.json
Ready to use Tokio Prompt Orchestrator?
Browse our complete directory of 33,000+ MCP servers, read setup guides for your editor, and start building with the Model Context Protocol.