MCP评估工具
MCP（模型上下文协议）服务器的论文等级评估工具，侧重于时间序列预测工具。
仓库结构
mcp_evaluation/
├── results/
│ ├── raw_audit_data.json # Full audit trail for all runs
│ ├── raw_audit_data_flattened.csv # One row per task run, ready for analysis
│ └── raw_audit_data_report.md # Markdown summary table for thesis write-up
├── __init__.py # Package declaration
├── config.py # Constants, logging, system prompt
├── models.py # Dataclasses and enums
├── utils.py # Shared helpers
├── xml_parser.py # evaluation_v2.xml → TaskCriteria
├── mcp_setup.py # Azure AD auth + MCP client setup
├── agent.py # Async LLM ↔ MCP agent loop
├── evaluator.py # Assertion checks + success scoring
├── reporting.py # CSV, task summary, Markdown report
├── main.py # Orchestration entrypoint (CLI)
├── evaluation.xml # Task specifications (place here)
├── requirements.txt
└── README.md设置
uv sync创建一个 .env 包含以下变量的文件：
TENANT_ID=...
CLIENT_ID=...
CLIENT_APP_SECRET=...
BACKEND_APP_ID=...
REMOTE_URL=...
LLM_BASE_URL=...
LLM_MODEL=...
API_KEY=...用法
python -m main --eval evaluation.xml --out raw_audit_data.json [--runs 10]输出：
raw_audit_data.json--所有运行的完整审计跟踪raw_audit_data_flattened.csv--每个任务运行一行，准备进行分析raw_audit_data_report.md--论文写作Markdown汇总表
模块概述
| 模块 | 责任 |
|---|---|
config.py | CONFIG_REQUIRED_VARS、日志记录， EVALUATION_SYSTEM_PROMPT |
models.py | 所有数据类： TaskCriteria, TaskResult, SuccessAssessment, ErrorEvent, … |
utils.py | _get_by_path, _classify_backend_error, _mean, _ci95_mean, … |
xml_parser.py | parse_evaluation_v2(xml_path) → list[TaskCriteria] |
mcp_setup.py | setup_mcp_client(), get_mcp_tools(), validate_tool_arguments() |
agent.py | agent_loop(...) → TaskResult |
evaluator.py | evaluate_task_success(...) → SuccessAssessment |
reporting.py | flatten_results_for_csv, compute_task_level_summary, render_markdown_summary |
main.py | run_evaluation(...), main() CLI入口点 |