vLLM MLX

Install via uv (recommended for isolation) or pip. For audio/TTS support, install with the optional audio extra and the espeak-ng system dependency.

uv tool install vllm-mlx
# Or with audio support:
pip install vllm-mlx[audio]
brew install espeak-ng

Launch the server with a model from mlx-community on Hugging Face. The model is downloaded automatically on first use. --continuous-batching improves throughput significantly.

vllm-mlx serve mlx-community/Llama-3.2-3B-Instruct-4bit --port 8000 --continuous-batching

Verify the server is running by sending a test request with the OpenAI Python SDK or curl. No real API key is needed — any string works.

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "default", "messages": [{"role": "user", "content": "Hello!"}]}'

Set the ANTHROPIC_BASE_URL and ANTHROPIC_API_KEY environment variables to redirect Claude Code to your local vllm-mlx server instead of Anthropic's cloud API.

export ANTHROPIC_BASE_URL=http://localhost:8000
export ANTHROPIC_API_KEY=not-needed
claude

To expose vllm-mlx as an MCP server that provides local model inference as a tool, add it to your claude_desktop_config.json.

{
  "mcpServers": {
    "vllm-mlx": {
      "command": "vllm-mlx",
      "args": ["serve", "mlx-community/Llama-3.2-3B-Instruct-4bit", "--port", "8000"],
      "env": {
        "ANTHROPIC_BASE_URL": "http://localhost:8000",
        "ANTHROPIC_API_KEY": "not-needed"
      }
    }
  }
}

For long-context models, add --ssd-cache-dir to spill the KV cache to disk. For reasoning models, add --reasoning-parser qwen3 or --reasoning-parser deepseek. Enable Prometheus metrics with --metrics.

vllm-mlx serve mlx-community/Qwen2.5-7B-Instruct-4bit \
  --port 8000 \
  --continuous-batching \
  --ssd-cache-dir /tmp/kvcache \
  --reasoning-parser qwen3 \
  --metrics