Https GitHub Com Steviec MCP Server Tester

MCP服务器测试器

MCP服务器测试工具是一款用于自动化测试MCP服务器的工具。

为什么选择这个工具？

构建可靠且始终如一、符合规范的MCP服务器，其难度远超预期：

• Inspector 工具有助于构建和测试初始服务器，但无法防止回归问题
• 这些工具在直接测试中表现完美，但在现实世界中，大型语言模型（LLMs）使用它们时却存在错误和不一致的情况
• 大多数MCP（可能是指某种特定领域的开发工具包或平台）SDK并不完全符合规范，而且很难知道还缺少哪些内容
• 规范迅速发展，使得紧跟最佳实践变得困难

该 mcp-server-tester 这是一个与实现无关的命令行界面（CLI）测试工具，提供了三个针对性的命令，有助于解决这些核心问题：

• tools - 运行测试以验证您的工具能够正确地与直接的MCP调用一起工作
• evals - 运行测试以验证大型语言模型（LLMs）能够一致地理解和正确使用您的工具
• compliance - 随着协议的演进，实现服务器规格的自动化验证 _（进行中）_

______________________________________________________________________

命令

tools - 直接MCP工具测试

测试您的工具是否能通过直接API调用正常工作：

npx mcp-server-tester tools tool-tests.yaml --server-config filesystem-server-config.json

样本输出：

✅ Tool Discovery: Found 3 tools (read_file, write_file, delete_file)
✅ Write file successfully
✅ Read file returns content
✅ Write and read workflow
❌ Handle invalid path - Expected error containing 'permission' but got 'access denied'
✅ Delete file cleanup

Results: 5/6 tests passed

evals - 大语言模型（LLM）集成测试

测试大型语言模型（LLMs）能否有效发现并使用您的工具：

npx mcp-server-tester evals eval-tests.yaml --server-config filesystem-server-config.json

示例输出：

🤖 Testing with claude-3-5-haiku-latest
✅ LLM can create and read files
✅ File manipulation workflow
✅ Error handling test
❌ Documentation query task: FAILED
    Prompt: "How do I search for configuration files in the project?"
    • Required tool 'search_docs' was not called (actual calls: list_files)
❌ Lists all available tools: FAILED
    Prompt: "Please list all available file management tools you have access to."
    • LLM judge failed: Score: 0.3/1.0 (threshold: 0.8). Assistant did not provide complete tool list.

Results: 3/5 tests passed

_注：需要 ANTHROPIC_API_KEY 环境变量。_

compliance - MCP规范合规性测试（进行中）

测试您的服务器是否符合规范：

npx mcp-server-tester compliance --server-config filesystem-server-config.json

看看 合规指令（在制品，WIP） 以下部分提供了详细信息和示例输出。

快速入门

1. 创建服务器配置 （filesystem-server-config.json)：

   {
     "mcpServers": {
       "filesystem": {
         "command": "node",
         "args": ["./filesystem-server.js"]
       }
     }
   }

1. 手动创建工具和评估测试文件:

tool-tests.yaml：

   tools:
     expected_tool_list: ['write_file']
     tests:
       - name: 'Write file successfully'
         tool: 'write_file'
         params: { path: '/tmp/test.txt', content: 'Hello world' }
         expect: { success: true }

eval-tests.yaml:

   evals:
     models: ['claude-3-5-haiku-latest']
     tests:
       - name: 'LLM can write files'
         prompt: 'Create a file at /tmp/greeting.txt with the content "Hello from Claude"'
         expected_tool_calls:
           required: ['write_file']
         response_scorers:
           - type: 'llm-judge'
             criteria: 'Did the assistant successfully create the file?'
             threshold: 0.8

请参阅 工具测试 并且 评估测试 包含全面句法示例的章节。

1. 使用代理自动创建工具和评估测试：

尝试使用这个提示，并将服务器配置信息替换为你自己的：

   Please create tool tests and eval tests for me to use with the mcp server tester tool.
   To see how to use it, run the tool's documentation and schema commands:

     `npx -y mcp-server-tester --help`

   My server config file is at ./filesystem-server-config.json. To know what tools you need to create tests for, run this command:

     `npx -y @modelcontextprotocol/inspector --cli --config filesystem-server-config.json --server filesystem-server --method tools/list`

   Please follow these steps:

   1. Create tool tests
     - Create a file called `tool-tests.yaml` that contains a single test for each tool. Follow these guidelines:
       - Do NOT force an individual test to pass; if the expected output is not returned, the test should fail
       - if there is a clear dependency between tool calls, you can chain them using the "calls" property
     - Run the tests and confirm that the syntax is correct and that each test runs (they do not have to pass)

   2. Create eval tests
     - Create a file called `eval-tests.yaml` with eval tests that will test the server's behavior. Follow these guidelines:
       - start with a few simple evals, and then build up to more complex ones
       - create between 5 and 10 eval tests
     - Run the tests and confirm that the syntax is correct and that each test runs (they do not have to pass)

   3. Provide a summary, which includes:
     - a list of the tools that are being tested and what you chose to test
     - a list of the eval tests and your reasoning for why you chose them
     - an explanation of how to run the tool tests and eval tests

1. 运行测试:

   # Run tools tests (fast, no API key needed)
   mcp-server-tester tools tool-tests.yaml --server-config filesystem-server-config.json

   # Run LLM evaluation tests (requires API key)
   export ANTHROPIC_API_KEY="your-key"
   mcp-server-tester evals eval-tests.yaml --server-config filesystem-server-config.json

工具测试

这个（或“该”） tools 命令通过直接的API调用测试您的MCP服务器工具。这种方法快速、无需API密钥，并且有助于在使用大型语言模型（LLMs）测试之前验证您的工具是否正常工作。

基本语法

tools:
  expected_tool_list: ['tool1', 'tool2'] # Optional: verify specific tools are available
  tests:
    - name: 'Test description'
      tool: 'tool_name'
      params: { param1: 'value1' }
      expect: { success: true }

单一工具测试

测试单个工具调用：

tools:
  tests:
    # Basic success test
    - name: 'Write file successfully'
      tool: 'write_file'
      params: { path: '/tmp/test.txt', content: 'hello world' }
      expect: { success: true }

    # Error handling test
    - name: 'Handle invalid path'
      tool: 'write_file'
      params: { path: '/invalid/path.txt', content: 'test' }
      expect:
        success: false
        error: { contains: 'permission denied' }

    # Result validation test
    - name: 'Read file returns content'
      tool: 'read_file'
      params: { path: '/tmp/existing.txt' }
      expect:
        success: true
        result: { contains: 'expected content' }

多步骤工作流测试

工具调用的测试序列：

tools:
  tests:
    - name: 'Write and read workflow'
      calls:
        - tool: 'write_file'
          params: { path: '/tmp/workflow.txt', content: 'test data' }
          expect: { success: true }

        - tool: 'read_file'
          params: { path: '/tmp/workflow.txt' }
          expect:
            success: true
            result: { equals: 'test data' }

        - tool: 'delete_file'
          params: { path: '/tmp/workflow.txt' }
          expect: { success: true }

期望类型

成功/失败：

expect: { success: true }   # Tool should succeed
expect: { success: false }  # Tool should fail

错误验证：

expect:
  success: false
  error: { contains: 'permission denied' } # Error message must contain text

结果验证：

expect:
  success: true
  result: { contains: 'partial text' }      # Result must contain text
  result: { equals: 'exact match' }         # Result must exactly match

评估测试

这个（或“该”） evals 命令测试旨在验证大型语言模型（LLMs）能够有效发现并使用您的工具。这需要一个 ANTHROPIC_API_KEY 并测试实际使用模式。

基本语法

evals:
  models: ['claude-3-5-haiku-latest']
  max_steps: 5 # Optional: limit conversation turns
  tests:
    - name: 'Test description'
      prompt: 'Task for the LLM to complete'
      expected_tool_calls:
        required: ['tool1', 'tool2']
        allowed: ['tool3']
      response_scorers:
        - type: 'llm-judge'
          criteria: 'Did the assistant complete the task?'

工具调用验证

控制大型语言模型（LLM）应使用的工具：

evals:
  tests:
    - name: 'LLM must use specific tools'
      prompt: 'Write a paragraph of poetry to file "poetry.md"'
      expected_tool_calls:
        required: ['write_file'] # Must use these tools
        allowed: ['read_file'] # Can use this tool, but all other tool uses will throw error

响应评分

评估大型语言模型（LLM）响应的质量：

正则表达式评分器：

response_scorers:
  - type: 'regex'
    pattern: 'success|completed|done' # Response must match pattern

大型语言模型（LLM）裁判评分员：

response_scorers:
  - type: 'llm-judge'
    criteria: 'Did the assistant successfully complete the file operations?'
    threshold: 0.8 # Score must be >= 0.8

高级选项

IDE模式验证

在你的测试文件顶部添加这一行，以启用VS Code等集成开发环境（IDE）中的自动模式验证和自动补全功能：

# yaml-language-server: $schema=https://raw.githubusercontent.com/steviec/mcp-server-tester/refs/heads/main/src/schemas/tests-schema.json

环境变量替换：

使用 ${VAR_NAME} 语法，用于将环境变量注入到您的测试配置中：

export API_BASE_URL="https://api.example.com"
export TEST_USER="alice"
mcp-server-tester tools test.yaml --server-config server.json

tools:
  tests:
    - name: 'Test ${TEST_USER} API access'
      tool: 'api_call'
      params:
        url: '${API_BASE_URL}/users/${TEST_USER}'
        method: 'GET'
      expect:
        success: true
        result: { contains: '${TEST_USER}' }

环境变量可在测试名称、参数、预期结果以及所有其他配置值中使用。缺少的变量将导致配置加载失败，并显示一条有用的错误信息。

调试模式：

查看完整的对话输出及评分详情：

mcp-server-tester evals eval-tests.yaml --server-config filesystem-server-config.json --debug

多种模型：

evals:
  models: ['claude-3-5-haiku-latest', 'claude-3-5-sonnet-latest']
  tests: [...]

复杂评估：

evals:
  tests:
    - name: 'Multi-step file management'
      prompt: |
        Create a file called notes.txt with "Meeting notes from today".
        Then read it back and confirm the content is correct.
        Finally, create a backup copy called notes-backup.txt.
      expected_tool_calls:
        required: ['write_file', 'read_file']
        allowed: ['write_file', 'read_file', 'copy_file']
      response_scorers:
        - type: 'llm-judge'
          criteria: 'Did the assistant create the original file, read it back, and create a backup?'
          threshold: 0.8
        - type: 'contains'
          text: 'Meeting notes from today'

统一测试文件：

你可以把两者结合起来 tools 并且 evals 为了方便起见，将各部分放在一个单独的测试文件中：

# Combined test file with both tools and evals
tools:
  expected_tool_list: ['write_file', 'read_file']
  tests:
    - name: 'Write file successfully'
      tool: 'write_file'
      params: { path: '/tmp/test.txt', content: 'Hello world' }
      expect: { success: true }

evals:
  models: ['claude-3-5-haiku-latest']
  tests:
    - name: 'LLM can write files'
      prompt: 'Create a file at /tmp/greeting.txt with content "Hello from Claude"'
      expected_tool_calls:
        required: ['write_file']
      response_scorers:
        - type: 'llm-judge'
          criteria: 'Did the assistant successfully create the file?'
          threshold: 0.8

在执行命令时，每个命令只会使用其各自对应的部分：

• mcp-server-tester tools unified-test.yaml 仅使用 tools: 部分；章节
• mcp-server-tester evals unified-test.yaml 仅使用 evals: 部分；章节

合规指令（在制品）

⚠️（警告符号，无具体文字含义，表示提醒或警告） 进行中的工作这个（或：这些，具体根据上下文确定） compliance 该命令尚未完全符合规范。

我并不是百分之百确信合规检查器应该内置在这个工具中。但它对我来说很有用，所以我会在这里包含它。它还远远不够完善。

npx mcp-server-tester compliance --server-config filesystem-server-config.json

示例输出：

🏥 MCP SERVER COMPLIANCE
Diagnosing server: Filesystem MCP (MCP Protocol 2024-11-05)
Started: 7/19/2025, 7:53:57 AM

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

📋 MCP SPECIFICATION COMPLIANCE SUMMARY

🔍 BASE PROTOCOL
   └─ Transport Layer      ✅  3/3 passed
   └─ JSON-RPC 2.0         ❌  0/2 passed

🔍 LIFECYCLE
   └─ Initialization       ✅  2/2 passed
   └─ Capability Negotiation ✅  1/1 passed
   └─ Protocol Version     ❌  0/1 passed

🔍 SERVER FEATURES
   └─ Tools                ❌  3/4 passed
   └─ Resources            skipped (not advertised)
   └─ Prompts              skipped (not advertised)

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

📊 OVERALL MCP COMPLIANCE: 80/100 (2 features skipped)

Server Capabilities: tools ✅ | prompts ⏭️ | resources ⏭️

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
⚠️  SPECIFICATION WARNINGS (4)

⚠️  SDK: JSON-RPC Protocol Compliance [Base Protocol - JSON-RPC 2.0]
   Expected: Full JSON-RPC 2.0 compliance
   Actual:   1 compliance issues detected
   → Fix: Check capability advertisement matches implementation
   → Fix: Verify method name spelling and casing

⚠️  SDK: Error Response Validation [Base Protocol - Error Handling]
   Expected: Proper JSON-RPC error codes and format
   Actual:   2 error handling issues detected
   → Fix: Ensure server returns proper JSON-RPC error codes
   → Fix: Verify that the server properly validates requests
   → Fix: Ensure error handling is implemented according to JSON-RPC spec

⚠️  Lifecycle: Protocol Version Negotiation
   Protocol version issues detected (2 findings, 4 validations)

⚠️  Server Features: Tools - Execution (tools/call)
   Expected: Proper tool execution or error handling
   Actual:   1 execution issues detected
   → Fix: Verify error response format compliance

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

📈 DETAILED COMPLIANCE BREAKDOWN
   base-protocol: 3/5 tests passed
   lifecycle: 7/8 tests passed
   server-features: 3/12 tests passed

🔗 SPECIFICATION REFERENCES
• MCP Specification: https://spec.modelcontextprotocol.io/
• JSON-RPC 2.0: https://www.jsonrpc.org/specification
• Error Codes: https://spec.modelcontextprotocol.io/specification/basic/error-handling/

许可证

麻省理工学院（MIT）