MCP服务器测试器
MCP服务器测试工具是一款用于自动化测试MCP服务器的工具。
为什么选择这个工具？
构建可靠且始终如一、符合规范的MCP服务器，其难度远超预期：
- Inspector 工具有助于构建和测试初始服务器，但无法防止回归问题
- 这些工具在直接测试中表现完美，但在现实世界中，大型语言模型（LLMs）使用它们时却存在错误和不一致的情况
- 大多数MCP（可能是指某种特定领域的开发工具包或平台）SDK并不完全符合规范，而且很难知道还缺少哪些内容
- 规范迅速发展，使得紧跟最佳实践变得困难
该 mcp-server-tester 这是一个与实现无关的命令行界面（CLI）测试工具，提供了三个针对性的命令，有助于解决这些核心问题：
tools- 运行测试以验证您的工具能够正确地与直接的MCP调用一起工作evals- 运行测试以验证大型语言模型（LLMs）能够一致地理解和正确使用您的工具compliance- 随着协议的演进，实现服务器规格的自动化验证 _（进行中）_
______________________________________________________________________
命令
tools - 直接MCP工具测试
测试您的工具是否能通过直接API调用正常工作：
npx mcp-server-tester tools tool-tests.yaml --server-config filesystem-server-config.json样本输出：
✅ Tool Discovery: Found 3 tools (read_file, write_file, delete_file)
✅ Write file successfully
✅ Read file returns content
✅ Write and read workflow
❌ Handle invalid path - Expected error containing 'permission' but got 'access denied'
✅ Delete file cleanup
Results: 5/6 tests passedevals - 大语言模型（LLM）集成测试
测试大型语言模型（LLMs）能否有效发现并使用您的工具：
npx mcp-server-tester evals eval-tests.yaml --server-config filesystem-server-config.json示例输出：
🤖 Testing with claude-3-5-haiku-latest
✅ LLM can create and read files
✅ File manipulation workflow
✅ Error handling test
❌ Documentation query task: FAILED
Prompt: "How do I search for configuration files in the project?"
• Required tool 'search_docs' was not called (actual calls: list_files)
❌ Lists all available tools: FAILED
Prompt: "Please list all available file management tools you have access to."
• LLM judge failed: Score: 0.3/1.0 (threshold: 0.8). Assistant did not provide complete tool list.
Results: 3/5 tests passed_注：需要 ANTHROPIC_API_KEY 环境变量。_
compliance - MCP规范合规性测试（进行中）
测试您的服务器是否符合规范：
npx mcp-server-tester compliance --server-config filesystem-server-config.json看看 合规指令（在制品，WIP） 以下部分提供了详细信息和示例输出。
快速入门
- 创建服务器配置 （
filesystem-server-config.json)：
{
"mcpServers": {
"filesystem": {
"command": "node",
"args": ["./filesystem-server.js"]
}
}
}- 手动创建工具和评估测试文件:
tool-tests.yaml：
tools:
expected_tool_list: ['write_file']
tests:
- name: 'Write file successfully'
tool: 'write_file'
params: { path: '/tmp/test.txt', content: 'Hello world' }
expect: { success: true }eval-tests.yaml:
evals:
models: ['claude-3-5-haiku-latest']
tests:
- name: 'LLM can write files'
prompt: 'Create a file at /tmp/greeting.txt with the content "Hello from Claude"'
expected_tool_calls:
required: ['write_file']
response_scorers:
- type: 'llm-judge'
criteria: 'Did the assistant successfully create the file?'
threshold: 0.8- 使用代理自动创建工具和评估测试：
尝试使用这个提示，并将服务器配置信息替换为你自己的：
Please create tool tests and eval tests for me to use with the mcp server tester tool.
To see how to use it, run the tool's documentation and schema commands:
`npx -y mcp-server-tester --help`
My server config file is at ./filesystem-server-config.json. To know what tools you need to create tests for, run this command:
`npx -y @modelcontextprotocol/inspector --cli --config filesystem-server-config.json --server filesystem-server --method tools/list`
Please follow these steps:
1. Create tool tests
- Create a file called `tool-tests.yaml` that contains a single test for each tool. Follow these guidelines:
- Do NOT force an individual test to pass; if the expected output is not returned, the test should fail
- if there is a clear dependency between tool calls, you can chain them using the "calls" property
- Run the tests and confirm that the syntax is correct and that each test runs (they do not have to pass)
2. Create eval tests
- Create a file called `eval-tests.yaml` with eval tests that will test the server's behavior. Follow these guidelines:
- start with a few simple evals, and then build up to more complex ones
- create between 5 and 10 eval tests
- Run the tests and confirm that the syntax is correct and that each test runs (they do not have to pass)
3. Provide a summary, which includes:
- a list of the tools that are being tested and what you chose to test
- a list of the eval tests and your reasoning for why you chose them
- an explanation of how to run the tool tests and eval tests- 运行测试:
# Run tools tests (fast, no API key needed)
mcp-server-tester tools tool-tests.yaml --server-config filesystem-server-config.json
# Run LLM evaluation tests (requires API key)
export ANTHROPIC_API_KEY="your-key"
mcp-server-tester evals eval-tests.yaml --server-config filesystem-server-config.json工具测试
这个（或“该”） tools 命令通过直接的API调用测试您的MCP服务器工具。这种方法快速、无需API密钥，并且有助于在使用大型语言模型（LLMs）测试之前验证您的工具是否正常工作。
基本语法
tools:
expected_tool_list: ['tool1', 'tool2'] # Optional: verify specific tools are available
tests:
- name: 'Test description'
tool: 'tool_name'
params: { param1: 'value1' }
expect: { success: true }单一工具测试
测试单个工具调用：
tools:
tests:
# Basic success test
- name: 'Write file successfully'
tool: 'write_file'
params: { path: '/tmp/test.txt', content: 'hello world' }
expect: { success: true }
# Error handling test
- name: 'Handle invalid path'
tool: 'write_file'
params: { path: '/invalid/path.txt', content: 'test' }
expect:
success: false
error: { contains: 'permission denied' }
# Result validation test
- name: 'Read file returns content'
tool: 'read_file'
params: { path: '/tmp/existing.txt' }
expect:
success: true
result: { contains: 'expected content' }多步骤工作流测试
工具调用的测试序列：
tools:
tests:
- name: 'Write and read workflow'
calls:
- tool: 'write_file'
params: { path: '/tmp/workflow.txt', content: 'test data' }
expect: { success: true }
- tool: 'read_file'
params: { path: '/tmp/workflow.txt' }
expect:
success: true
result: { equals: 'test data' }
- tool: 'delete_file'
params: { path: '/tmp/workflow.txt' }
expect: { success: true }期望类型
成功/失败：
expect: { success: true } # Tool should succeed
expect: { success: false } # Tool should fail错误验证：
expect:
success: false
error: { contains: 'permission denied' } # Error message must contain text结果验证：
expect:
success: true
result: { contains: 'partial text' } # Result must contain text
result: { equals: 'exact match' } # Result must exactly match评估测试
这个（或“该”） evals 命令测试旨在验证大型语言模型（LLMs）能够有效发现并使用您的工具。这需要一个 ANTHROPIC_API_KEY 并测试实际使用模式。
基本语法
evals:
models: ['claude-3-5-haiku-latest']
max_steps: 5 # Optional: limit conversation turns
tests:
- name: 'Test description'
prompt: 'Task for the LLM to complete'
expected_tool_calls:
required: ['tool1', 'tool2']
allowed: ['tool3']
response_scorers:
- type: 'llm-judge'
criteria: 'Did the assistant complete the task?'工具调用验证
控制大型语言模型（LLM）应使用的工具：
evals:
tests:
- name: 'LLM must use specific tools'
prompt: 'Write a paragraph of poetry to file "poetry.md"'
expected_tool_calls:
required: ['write_file'] # Must use these tools
allowed: ['read_file'] # Can use this tool, but all other tool uses will throw error响应评分
评估大型语言模型（LLM）响应的质量：
正则表达式评分器：
response_scorers:
- type: 'regex'
pattern: 'success|completed|done' # Response must match pattern大型语言模型（LLM）裁判评分员：
response_scorers:
- type: 'llm-judge'
criteria: 'Did the assistant successfully complete the file operations?'
threshold: 0.8 # Score must be >= 0.8高级选项
IDE模式验证
在你的测试文件顶部添加这一行，以启用VS Code等集成开发环境（IDE）中的自动模式验证和自动补全功能：
# yaml-language-server: $schema=https://raw.githubusercontent.com/steviec/mcp-server-tester/refs/heads/main/src/schemas/tests-schema.json环境变量替换：
使用 ${VAR_NAME} 语法，用于将环境变量注入到您的测试配置中：
export API_BASE_URL="https://api.example.com"
export TEST_USER="alice"
mcp-server-tester tools test.yaml --server-config server.jsontools:
tests:
- name: 'Test ${TEST_USER} API access'
tool: 'api_call'
params:
url: '${API_BASE_URL}/users/${TEST_USER}'
method: 'GET'
expect:
success: true
result: { contains: '${TEST_USER}' }环境变量可在测试名称、参数、预期结果以及所有其他配置值中使用。缺少的变量将导致配置加载失败，并显示一条有用的错误信息。
调试模式：
查看完整的对话输出及评分详情：
mcp-server-tester evals eval-tests.yaml --server-config filesystem-server-config.json --debug多种模型：
evals:
models: ['claude-3-5-haiku-latest', 'claude-3-5-sonnet-latest']
tests: [...]复杂评估：
evals:
tests:
- name: 'Multi-step file management'
prompt: |
Create a file called notes.txt with "Meeting notes from today".
Then read it back and confirm the content is correct.
Finally, create a backup copy called notes-backup.txt.
expected_tool_calls:
required: ['write_file', 'read_file']
allowed: ['write_file', 'read_file', 'copy_file']
response_scorers:
- type: 'llm-judge'
criteria: 'Did the assistant create the original file, read it back, and create a backup?'
threshold: 0.8
- type: 'contains'
text: 'Meeting notes from today'统一测试文件：
你可以把两者结合起来 tools 并且 evals 为了方便起见，将各部分放在一个单独的测试文件中：
# Combined test file with both tools and evals
tools:
expected_tool_list: ['write_file', 'read_file']
tests:
- name: 'Write file successfully'
tool: 'write_file'
params: { path: '/tmp/test.txt', content: 'Hello world' }
expect: { success: true }
evals:
models: ['claude-3-5-haiku-latest']
tests:
- name: 'LLM can write files'
prompt: 'Create a file at /tmp/greeting.txt with content "Hello from Claude"'
expected_tool_calls:
required: ['write_file']
response_scorers:
- type: 'llm-judge'
criteria: 'Did the assistant successfully create the file?'
threshold: 0.8在执行命令时，每个命令只会使用其各自对应的部分：
mcp-server-tester tools unified-test.yaml仅使用tools:部分；章节mcp-server-tester evals unified-test.yaml仅使用evals:部分；章节
合规指令（在制品）
⚠️（警告符号，无具体文字含义，表示提醒或警告） 进行中的工作这个（或：这些，具体根据上下文确定） compliance 该命令尚未完全符合规范。我并不是百分之百确信合规检查器应该内置在这个工具中。但它对我来说很有用，所以我会在这里包含它。它还远远不够完善。
npx mcp-server-tester compliance --server-config filesystem-server-config.json示例输出：
🏥 MCP SERVER COMPLIANCE
Diagnosing server: Filesystem MCP (MCP Protocol 2024-11-05)
Started: 7/19/2025, 7:53:57 AM
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
📋 MCP SPECIFICATION COMPLIANCE SUMMARY
🔍 BASE PROTOCOL
└─ Transport Layer ✅ 3/3 passed
└─ JSON-RPC 2.0 ❌ 0/2 passed
🔍 LIFECYCLE
└─ Initialization ✅ 2/2 passed
└─ Capability Negotiation ✅ 1/1 passed
└─ Protocol Version ❌ 0/1 passed
🔍 SERVER FEATURES
└─ Tools ❌ 3/4 passed
└─ Resources skipped (not advertised)
└─ Prompts skipped (not advertised)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
📊 OVERALL MCP COMPLIANCE: 80/100 (2 features skipped)
Server Capabilities: tools ✅ | prompts ⏭️ | resources ⏭️
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
⚠️ SPECIFICATION WARNINGS (4)
⚠️ SDK: JSON-RPC Protocol Compliance [Base Protocol - JSON-RPC 2.0]
Expected: Full JSON-RPC 2.0 compliance
Actual: 1 compliance issues detected
→ Fix: Check capability advertisement matches implementation
→ Fix: Verify method name spelling and casing
⚠️ SDK: Error Response Validation [Base Protocol - Error Handling]
Expected: Proper JSON-RPC error codes and format
Actual: 2 error handling issues detected
→ Fix: Ensure server returns proper JSON-RPC error codes
→ Fix: Verify that the server properly validates requests
→ Fix: Ensure error handling is implemented according to JSON-RPC spec
⚠️ Lifecycle: Protocol Version Negotiation
Protocol version issues detected (2 findings, 4 validations)
⚠️ Server Features: Tools - Execution (tools/call)
Expected: Proper tool execution or error handling
Actual: 1 execution issues detected
→ Fix: Verify error response format compliance
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
📈 DETAILED COMPLIANCE BREAKDOWN
base-protocol: 3/5 tests passed
lifecycle: 7/8 tests passed
server-features: 3/12 tests passed
🔗 SPECIFICATION REFERENCES
• MCP Specification: https://spec.modelcontextprotocol.io/
• JSON-RPC 2.0: https://www.jsonrpc.org/specification
• Error Codes: https://spec.modelcontextprotocol.io/specification/basic/error-handling/许可证
麻省理工学院（MIT）