Eval Action

MCP服务器评估操作

一个可重用的GitHub Action，可以运行 法学硕士作为评委评价 针对任何MCP服务器。在YAML文件中定义你的测试用例，该操作将启动你的服务器，对每个提示运行一个AI代理，用LLM法官对响应进行评分，并可以选择用正确的参数验证调用正确工具的代理。

适用于任何MCP服务器（stdio或remote）以及上可用的任何型号 开放路由.

运作原理

eval_cases.yaml ──► Agent runs prompt ──► LLM judge scores response ──► Report
                         │                        │
                         │                        ├── Rubric score (GEval)
                         ▼                        └── Tool assertions (programmatic)
                    MCP Server

1. 对于每一个 (case × model × system_prompt) 通过组合，创建了一个新的MCP代理
2. 代理对您的MCP服务器运行提示（每种情况都有自己的服务器实例）
3. DeepEval的案例 根据你的评分标准对代理人的反应进行评分
4. 如果 required_tools 如果定义了，则该操作将使用预期的参数验证调用了正确工具的代理
5. 案件只有在以下情况下才能通过 两者 量规得分达到阈值 和 所有工具断言都通过
6. 结果以JSON和markdown报告的形式输出，准备作为PR评论发布

快速开始

1.创建 evals/eval_cases.yaml 在您的repo中

judge_model: openai/gpt-4o

models:
  - anthropic/claude-sonnet-4
  - openai/gpt-4o-mini

system_prompts:
  neutral: "You are a helpful assistant."

cases:
  - id: basic_query
    prompt: "What are the top items in the database?"
    rubric: |
      The response should list items from the database.
      Each item should include a name and relevant details.
    threshold: 0.7

2.添加工作流

name: MCP Server Evals

on:
  pull_request:
  push:
    branches: [main]
  workflow_dispatch:

permissions:
  pull-requests: write
  contents: read

jobs:
  evals:
    runs-on: ubuntu-latest
    timeout-minutes: 30
    steps:
      - uses: actions/checkout@v4

      # Install your MCP server's dependencies
      - uses: actions/setup-python@v5
        with:
          python-version: "3.11"
      - uses: astral-sh/setup-uv@v5
      - run: uv pip install --system -r requirements.txt

      # Run evals
      - uses: mcp-use/eval-action@v1.4
        id: evals
        with:
          server_config: |
            {
              "command": "python",
              "args": ["-m", "my_mcp_server", "--transport", "stdio"],
              "env": {
                "API_KEY": "${{ secrets.API_KEY }}"
              }
            }
          eval_cases: evals/eval_cases.yaml
          openrouter_api_key: ${{ secrets.OPENROUTER_API_KEY }}

      # Post results as a sticky PR comment
      - uses: marocchino/sticky-pull-request-comment@v2
        if: always() && github.event_name == 'pull_request'
        with:
          header: mcp-evals
          path: ${{ steps.evals.outputs.report_md }}

      # Also show in GitHub Actions summary
      - run: cat ${{ steps.evals.outputs.report_md }} >> "$GITHUB_STEP_SUMMARY"
        if: always()

      # Upload artifacts for later inspection
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: eval-results
          path: |
            ${{ steps.evals.outputs.results_json }}
            ${{ steps.evals.outputs.report_md }}
          retention-days: 30

3.添加您的OpenRouter API密钥

首选 设置→ 秘密与变量→ 行动 并添加 OPENROUTER_API_KEY.

就是这样。每个公关现在都会收到一份评估报告作为评论。

Eval案例YAML参考

这 eval_cases.yaml 该文件定义了所有内容：要测试哪些模型，运行哪些提示，以及如何对结果进行评分。

# ── Judge configuration ──────────────────────────────────────────────────────
# The model that scores agent responses. Can be overridden with
# the EVAL_JUDGE_MODEL env var.
judge_model: openai/gpt-4o

# ── Models under test ────────────────────────────────────────────────────────
# Each case runs once per model. Use OpenRouter model IDs.
# Can be overridden with EVAL_MODELS env var (comma-separated).
models:
  - anthropic/claude-sonnet-4
  - openai/gpt-4o-mini

# ── System prompts ───────────────────────────────────────────────────────────
# Each case runs once per prompt. Use this to compare how the agent
# behaves with different instructions.
# The placeholder {today} is replaced with the current date (YYYY-MM-DD).
system_prompts:
  neutral: "You are a helpful assistant."
  domain: "You are a domain expert. Today is {today}. Use the available tools."

# ── Test cases ───────────────────────────────────────────────────────────────
cases:
  - id: my_test_case          # Unique identifier (used in reports and --filter)
    prompt: "Ask the agent something"
    rubric: |                  # What a good response looks like (scored by LLM judge)
      The response should contain relevant information.
      The response should be well-structured.
    required_tools:            # (Optional) Tools the agent must call
      - lookup_item
      - name: search_records
        args:
          category: "electronics"
    threshold: 0.7             # Minimum GEval score to pass (0.0 – 1.0)

使用3个案例×2个模型×2个提示，您将获得12次eval运行。

评分标准

这个量规是一个简单的文本描述，描述了一个好的回应是什么样子的。法学硕士法官宣读代理人的 最终文本响应 并根据评分标准对其进行评分。保持评价标准侧重于反应的可观察质量：

rubric: |
  The response should list items ranked by relevance.
  Each entry should include a name and a brief description.
  The response should not include internal database IDs.

法官确实如此 不 请查看工具调用——仅查看最终响应。为了验证工具的使用情况，请使用 required_tools （见下文）。

工具断言

工具断言验证代理在执行过程中调用的特定工具。它们被检查过了 以编程方式 根据代理人的谈话记录，没有涉及法学硕士。

案件只有在以下情况下才能通过 两者 量规得分达到阈值 和 所有工具断言都通过。

简单的形式——只需检查工具是否被调用

required_tools:
  - resolve_category
  - search_records

参数匹配

required_tools:
  - name: lookup_item
    args:
      query: { contains: "widget" }   # case-insensitive substring
  - name: search_records
    args:
      region: "us-east"               # exact match (case-insensitive)

如果该工具被多次调用（例如。， search_records 随着 region: "us-east" 然后与 region: "eu-west")，只要满足以下条件，该断言就成立 至少一个电话 与预期参数匹配。

参数匹配模式

结合简单和详细的表格

您可以在同一列表中混合这两种形式：

required_tools:
  - resolve_category                     # just check it was called
  - name: lookup_item
    args:
      query: { contains: "widget" }      # check name + args
  - name: search_records
    args:
      region: "us-east"
  - name: search_records
    args:
      region: "eu-west"

行动输入

行动成果

服务器配置

Stdio服务器（子流程）

该操作将服务器作为每个eval案例的子流程启动：

{
  "command": "python",
  "args": ["-m", "my_mcp_server", "--transport", "stdio"],
  "env": {
    "DATABASE_URL": "postgres://...",
    "API_KEY": "secret"
  }
}

远程服务器（HTTP）

连接到已运行的MCP服务器：

{
  "url": "https://my-server.example.com/mcp"
}

环境变量覆盖

这些env变量覆盖了相应的YAML字段：

报告格式

生成的降价报告包括：

汇总表 每次eval运行一行：

这 工具 仅当至少有一个案例满足以下条件时，才会显示该列 required_tools 定义。

可折叠细节 对于每次运行，包含：

• 原始查询
• 评判分数和推理
• 工具断言结果（如适用）——每个工具通过/失败，预期参数与实际参数
• 完整的代理响应

分数徽章

### CLI选项

python run_evals.py \ --server-config JSON # MCP server config (required) --eval-cases PATH # Path to eval_cases.yaml (required) --output PATH # Output JSON path (default: eval-results.json) --filter STRING # Filter cases by id substring --max-steps N # Max agent steps per case (default: 30) --parallel # Run in parallel (default) --no-parallel # Run sequentially

## 结果JSON模式

每个条目 `eval-results.json`:

{ "case_id": "specific_lookup", "model": "anthropic/claude-sonnet-4", "prompt_name": "neutral", "success": true, "rubric_passed": true, "tools_passed": true, "input": "Find details about the Widget Pro in the US store.", "actual_output": "Here are the details for Widget Pro...", "metrics": [ { "name": "Response Quality", "score": 0.85, "reason": "The response covers all requested details...", "success": true } ], "tool_calls": [ { "name": "lookup_item", "args": { "query": "Widget Pro" } }, { "name": "search_records", "args": { "region": "us-east", "item": "Widget Pro" } } ], "tool_assertions": { "passed": true, "checks": [ { "tool": "lookup_item", "passed": true, "reason": "called with matching args", "expected_args": { "query": { "contains": "widget" } } }, { "tool": "search_records", "passed": true, "reason": "called with matching args", "expected_args": { "region": "us-east" } } ] }, "duration_s": 12.3, "total_duration_s": 45.0 }