MCProbe
MCP（模型上下文协议）服务器的会话测试框架。MCProbe验证MCP服务器是否为LLM代理提供了足够的信息，以便使用合成用户和LLM判断正确回答现实世界的问题。
特性
- 基于场景的测试：使用合成用户角色和评估标准在YAML中定义测试场景
- 合成用户：具有不同耐心水平、专业知识和沟通风格的可配置用户角色
- 自动评估：基于LLM的法官评估正确性、工具使用和效率
- 多种代理类型：使用MCP工具测试简单的LLM代理或Gemini ADK代理
- 提示和模式跟踪：通过自动更改检测跨运行跟踪代理提示和MCP工具模式
- 趋势分析：跟踪测试性能随时间的变化并检测回归
- 缺陷检测：自动识别不一致的测试
- CI/CD集成：pytest插件、JUnitXML报告、GitHub操作支持
- 场景生成：从MCP服务器工具模式自动生成测试场景
快速开始
安装
pip install mcprobe对于Gemini ADK代理支持：
pip install mcprobe[adk]先决条件
MCProbe要求合成用户和裁判使用LLM提供商。选择一个：
选项1：Ollama（当地，免费）
# Install Ollama (macOS/Linux)
curl -fsSL https://ollama.com/install.sh | sh
# Start Ollama and pull a model
ollama serve
ollama pull llama3.2选项2:OpenAI（云）
# Set your API key
export OPENAI_API_KEY="sk-your-key-here"OpenAI提供商还与Azure OpenAI、vLLM、LiteLLM和其他OpenAI兼容服务一起工作。
创建场景
创建 test-scenario.yaml:
name: Weather Query Test
description: Test that the agent can answer weather questions
synthetic_user:
persona: A user planning a weekend trip
initial_query: What's the weather like in San Francisco this weekend?
max_turns: 5
evaluation:
correctness_criteria:
- Agent provides weather information for San Francisco
- Response includes temperature or conditions运行测试
与Ollama：
mcprobe run test-scenario.yaml使用OpenAI：
mcprobe run test-scenario.yaml --provider openai --model gpt-4使用配置文件：
创建 mcprobe.yaml:
llm:
provider: ollama
model: llama3.2
base_url: http://localhost:11434
# Optional: Track MCP schema changes
mcp_server:
command: "npx @modelcontextprotocol/server-weather"
# OR for HTTP-based MCP server:
# url: "http://localhost:8080/mcp"
# OR with authentication:
# url: "http://localhost:8080/mcp"
# headers:
# Authorization: "Bearer ${API_TOKEN:-dev}"然后运行：
mcprobe run test-scenario.yaml查看结果
Result: PASSED (score: 0.85)
Reasoning: The agent successfully provided weather information...文档
CLI命令
# Run scenarios
mcprobe run scenarios/ # Run all scenarios in directory
mcprobe run scenario.yaml -v # Run with verbose output
# Use different providers
mcprobe run scenario.yaml --provider ollama --model llama3.2
mcprobe run scenario.yaml --provider openai --model gpt-4
# Use configuration file
mcprobe run scenario.yaml --config mcprobe.yaml
mcprobe run scenario.yaml -c mcprobe.yaml
# Generate scenarios from MCP server
mcprobe generate-scenarios --server "npx @example/weather-mcp" -o ./scenarios
# Generate reports
mcprobe report --format html --output report.html
# Analyze trends
mcprobe trends --window 20
mcprobe flaky --fail-on-flaky
# Validate scenarios
mcprobe validate scenarios/
# List available providers
mcprobe providerspytest集成
MCProbe包括一个用于无缝测试集成的pytest插件：
# Run scenarios as pytest tests
pytest scenarios/ -v
# Use config file
pytest scenarios/ --mcprobe-config mcprobe.yaml
# Override config with CLI options
pytest scenarios/ --mcprobe-provider openai --mcprobe-model gpt-4
# Save results for analysis
pytest scenarios/ --mcprobe-save-results
# Filter by tags
pytest scenarios/ -m smoke示例场景
name: Multi-City Weather Comparison
description: Test comparing weather across multiple cities
synthetic_user:
persona: A business traveler deciding between meeting locations
initial_query: Compare the weather in New York, Chicago, and Miami for next Tuesday
max_turns: 8
clarification_behavior:
known_facts:
- Meeting is scheduled for next Tuesday
- Prefer outdoor lunch if weather permits
traits:
patience: medium
verbosity: concise
expertise: intermediate
evaluation:
correctness_criteria:
- Provides weather for all three cities
- Includes temperature information
- Makes a recommendation based on weather
failure_criteria:
- Provides weather for wrong cities
- Gives conflicting information
tool_usage:
required_tools:
- get_weather
efficiency:
max_tool_calls: 5发展
# Clone and install
git clone https://github.com/Liquescent-Development/mcprobe.git
cd mcprobe
uv venv
source .venv/bin/activate
uv sync --all-extras
# Run tests
pytest tests/unit/ -v
# Lint and type check
ruff check src/
mypy src/许可证
AGPL-3.0许可证-请参阅 许可证 了解详情。
链接
-