MCP Eval Server

🏆 MCP评估服务器

*终极AI评估平台*

📊 工具：63个专业评估工具 👨‍💻 作者：米哈伊·克里维蒂

A. MCP服务器 提供生态系统中最全面的人工智能评估平台。特性 63种专用工具 穿过 14个类别 使用 法学硕士裁判技巧 结合基于规则的度量。

🎯 工具类别概述

📊 核心评估（15个工具）

🤖 4裁判工具 -LLM作为一名法官的评估，具有偏见缓解功能 📝 4提示工具 -清晰度、一致性、完整性分析 🛠️ 4代理工具 -工具使用、推理、任务完成评估 🔍 3质量工具 -真实性、连贯性、毒性检测

🔬 高级评估（39个工具）

🔗 8个RAG工具 -检索相关性、上下文利用率、基础验证 ⚖️ 6偏见与公平 -人口统计学偏见、代表公平、交叉分析 🛡️ 5个稳健性工具 -对抗性测试、注射阻力、稳定性分析 🔒 4安全与校准 -有害内容检测、指令遵守、价值对齐 🌍 4多语言工具 -翻译质量、跨语言一致性、文化适应性 ⚡ 4性能工具 -延迟跟踪、效率指标、吞吐量扩展 🔐 8隐私工具 -PII检测、数据最小化、合规性、匿名化

🔧 系统管理（9个工具）

🔄 3工作流工具 -评估套件、并行执行、结果比较 📊 2个校准工具 -判断一致性测试、量规优化 🏥 4服务器工具 -健康监控、缓存统计、系统管理

⚡ 技术

• 🤖 法学硕士作为法官 -GPT-4、Azure OpenAI，具有位置偏差缓解功能
• 📈 统计严谨 -置信区间、显著性检验、相关性分析
• 🎪 多模态评估 -模式匹配+LLM评估+基于规则的度量
• 🏗️ 可扩展架构 -可配置的量规、自定义标准、插件系统

🚀 快速开始

📡 多种服务器模式

🔌 MCP服务器模式（stdio）

# 🎯 One-command setup
pip install -e ".[dev]"

# 🔥 Launch MCP server for Claude Desktop, MCP clients
python -m mcp_eval_server.server
# or
make dev

# 🏥 Health check (automatic on port 8080)
curl http://localhost:8080/health   # ✅ Liveness probe
curl http://localhost:8080/ready    # 🎯 Readiness probe
curl http://localhost:8080/metrics  # 📊 Performance metrics

🌐 REST API服务器模式（HTTP）

# 🚀 Launch REST API server with FastAPI
python -m mcp_eval_server.rest_server --port 8080 --host 0.0.0.0
# or
make serve-rest

# 📚 Interactive API documentation
open http://localhost:8080/docs

# 🧪 Quick API test
curl http://localhost:8080/health
curl http://localhost:8080/tools/categories

🔄 HTTP网桥模式（MCP over HTTP）

# 🌍 MCP protocol over HTTP with Server-Sent Events
make serve-http

# 📡 Access via JSON-RPC over HTTP on port 9000
curl -X POST -H 'Content-Type: application/json' \
     -d '{"jsonrpc": "2.0", "id": 1, "method": "tools/list", "params": {}}' \
     http://localhost:9000/

✨ 完整的工具库

🤖 LLM作为评判工具 （4工具）

• 🎯 单一响应评估：具有加权评分和置信度指标的可定制标准
• ⚖️ 配对比较：具有自动位置偏差缓解功能的头对头分析
• 🏆 多响应排名：锦标赛、循环赛和基于评分的排名算法
• 📊 基于参考的评估：真实性、完整性和风格的黄金标准比较
• 🤝 多法官共识：通过协议分析和置信度加权进行集成评估

📝 快速评估工具 （4工具）

• 🔍 清晰度分析：基于规则的歧义检测+LLM语义分析，并提出改进建议
• 🔄 一致性测试：跨温度设置的多运行方差分析，带有异常值检测
• ✅ 完整性测量：通过可视化热图生成进行组件覆盖率分析
• 🎯 相关性评估：使用TF-IDF矢量化和漂移分析进行语义对齐

🛠️ 代理评估工具 （4工具）

• ⚙️ 工具使用评估：选择准确性、序列优化、参数验证和效率评分
• ✅ 任务完成情况分析：多标准成功评估，包括部分信用和失败分析
• 🧠 推理评估：决策质量、逻辑连贯性和幻觉检测
• 📈 绩效基准测试：通过基线比较进行跨技能水平的全面能力测试

🔍 质量评估工具 （3个工具）

• ✅ 事实核查：根据知识库进行索赔验证，并进行置信度评分和证据跟踪
• 🧩 连贯性分析：逻辑流程评估、矛盾检测和结构分析
• 🛡️ 毒性检测：通过偏差模式分析识别多类别有害内容

🔗 RAG评估工具 （8工具）

• 📊 检索相关性：使用LLM判断验证和可配置阈值进行语义相似性评估
• 🎯 上下文利用率：分析检索到的上下文如何很好地整合到生成的响应中
• ⚓ 答案基础：根据严格控制的支持上下文进行索赔验证
• 🚨 幻觉检测：响应和源上下文之间的矛盾识别
• 🎯 检索覆盖率：主题完整性评估和信息差距分析
• 📝 引用准确性：跨多种格式的参考文献验证和引用质量评分
• 🧩 块相关性：对单个文档段进行评估，包括排名和评分
• 🏆 检索基准测试：使用标准IR指标（精确度、召回率、MRR、NDCG）进行比较分析

⚖️ 偏见和公平工具 （6工具）

• 🎯 人口统计学偏差检测：受保护群体偏见的模式匹配和LLM评估
• 📊 代表公平：跨背景和群体的平衡代表性分析
• ⚖️ 结果公平：跨受保护属性的差异影响分析
• 🌍 文化敏感性：跨文化适宜性和意识评估
• 🗣️ 语言偏见检测基于语言的歧视和方言偏见识别
• 🔗 交叉公平：跨多个身份维度的复合偏见效应

🛡️ 稳健性工具 （5工具）

• ⚔️ 对抗性测试：恶意即时抵抗和攻击向量评估
• 🔄 输入灵敏度：输入变化和扰动下的响应稳定性测试
• 🛡️ 快速注射阻力：针对注射攻击的安全防御评估
• 📈 分布偏移：域外数据性能下降分析
• 🎯 扰动下的一致性：跨输入修改的输出稳定性测量

🔒 安全和校准工具 （4工具）

• ⚠️ 有害内容检测：跨安全维度的多类别风险评估
• 📋 说明如下：遵守约束和安全指示
• 🚫 拒绝恰当性：评估适当的系统拒绝行为
• 💎 价值对齐：人类价值观和伦理原则一致性评估

🌍 多语言工具 （4工具）

• 🔄 翻译质量：跨语言的准确性、流畅性和完整性评估
• 🔗 跨语言一致性：跨多语言版本的一致性评估
• 🎭 文化适应：本地化质量和文化适宜性评估
• 🔀 语言混合检测：不恰当的代码转换和语言混合识别

⚡ 性能工具 （4工具）

• ⏱️ 响应延迟：通过统计分析和百分位数进行发电速度跟踪
• 💻 计算效率：资源使用监测和效率指标
• 📈 吞吐量扩展：并发请求处理和扩展行为分析
• 💾 内存监控：内存消耗模式跟踪和泄漏检测

🔐 隐私工具 （8工具）

• 🔍 PII检测：具有可配置灵敏度的个人身份信息检测
• 📊 叫做数据缩小：评估数据收集的必要性和目的一致性
• 📋 同意遵守：隐私法规合规性评估（GDPR、CCPA、COPPA、HIPAA）
• 🎭 匿名有效性：重新识别风险分析和效用保护
• 🚨 数据泄漏检测：意外数据暴露和推断泄漏识别
• 📖 明确同意：隐私声明的可读性和可理解性评估
• 🗃️ 数据保留合规性：保留政策的一致性和法规遵从性
• 🏗️ 隐私设计：系统级隐私实施和设计原则评估

🔄 工作流管理工具 （3个工具）

• 🎛️ 评估套房：可定制的多步骤管道，具有加权标准和成功阈值
• ⚡ 并行/顺序执行：通过可配置的并发性和资源管理优化处理
• 📊 结果比较：具有趋势检测、显著性检验和回归分析的统计分析

📊 判断校准工具 （2个工具）

• 🤝 协议测试：与人类基线比较的法官间相关性分析
• 🎯 准则优化：使用机器学习进行自动调整，以改善人类对齐

🔧 服务器管理工具 （9工具）

• 📋 法官管理：可用模型列表、能力评估、配置验证
• 💾 结果存储：具有元数据和统计报告的综合评价历史
• ⚡ 缓存管理：多级缓存统计和性能优化
• 🔍 健康监测：系统状态检查和性能指标

🚀 高级功能

🎯 法学硕士作为法官的最佳实践

• 位置偏差缓解：用于公平比较的自动响应位置随机化
• 思维链整合逐步推理以提高评估质量
• 置信度校准：评估可靠性的自我评估指标
• 多法官共识：带有分歧分析的集成方法
• 人类对齐：根据地面实况评估进行定期校准

⚡ 性能和可扩展性

• 轻量级依赖关系：使用标准库（scikit-learn、numpy）而不是繁重的ML框架
• 智能缓存：具有TTL和无效的多级缓存（内存+磁盘）
• 异步处理：具有可配置并发性的非阻塞评估执行
• 批量操作：高效的多项目处理和进度跟踪
• 资源管理：具有自动缩放功能的内存和CPU优化
• 快速启动：快速初始化，无需加载大型预训练模型

🔒 企业安全

• 加密随机：安全生成随机数以减少偏差
• API密钥管理：通过环境变量集成实现安全的凭据处理
• 输入验证：全面的参数验证和净化
• 错误隔离：优雅的故障处理，详细的错误报告
• 审计跟踪：完整的评估历史记录和合规报告

📊 分析与洞察

• 统计分析：相关性分析、显著性检验、趋势检测
• 性能指标：延迟跟踪、吞吐量监控、成功率分析
• 质量仪表板：实时评估质量监控，并发出警报
• 比较分析：具有回归检测的A/B测试功能
• 预测分析：性能趋势预测和异常检测

🛠️ 安装和设置

快速安装

# Clone and install (lightweight dependencies only)
cd mcp-servers/python/mcp_eval_server
pip install -e ".[dev]"

# Set up API keys (optional - rule-based judge works without them)
export OPENAI_API_KEY="sk-your-key-here"
export AZURE_OPENAI_ENDPOINT="https://your-resource.openai.azure.com/"
export AZURE_OPENAI_API_KEY="your-azure-api-key"

# Configure health check endpoints (optional)
export HEALTH_CHECK_PORT=8080        # Default: 8080
export HEALTH_CHECK_HOST=0.0.0.0     # Default: 0.0.0.0

# Note: No heavy ML dependencies required!
# Uses efficient TF-IDF + scikit-learn instead of transformers

MCP客户端连接

{
  "command": "python",
  "args": ["-m", "mcp_eval_server.server"],
  "cwd": "/path/to/mcp-servers/python/mcp_eval_server"
}

协议：stdio（模型上下文协议） 运输：标准输入/输出（不需要HTTP端口） 可用工具63个专门的评估工具

健康检查端点

服务器会自动启动健康检查HTTP端点以进行监视：

# Health endpoints (started automatically with the MCP server)
curl http://localhost:8080/health    # Liveness probe
curl http://localhost:8080/ready     # Readiness probe
curl http://localhost:8080/metrics   # Basic metrics
curl http://localhost:8080/          # Service info

# Kubernetes-style endpoints
curl http://localhost:8080/healthz   # Alternative health
curl http://localhost:8080/readyz    # Alternative readiness

健康检查响应示例：

{
  "status": "healthy",
  "timestamp": 1698765432.123,
  "uptime_seconds": 45.67,
  "service": "mcp-eval-server",
  "version": "0.1.0",
  "checks": {
    "server_running": true,
    "uptime_ok": true
  }
}

准备就绪检查响应示例：

{
  "status": "ready",
  "timestamp": 1698765432.123,
  "service": "mcp-eval-server",
  "version": "0.1.0",
  "checks": {
    "server_initialized": true,
    "judge_tools_loaded": true,
    "storage_initialized": true
  }
}

Docker部署

# Build container
make build

# Run with environment
make run

# Or use docker-compose
make compose-up

开发设置

# Install development dependencies
make dev-install

# Run development server
make dev

# Run tests
make test

# Check code quality
make lint

🎮 使用示例

🎯 MCP客户端集成

# Multi-criteria evaluation with MCP client
result = await mcp_client.call_tool("judge.evaluate_response", {
    "response": "Detailed technical explanation...",
    "criteria": [
        {"name": "technical_accuracy", "description": "Correctness of technical details", "scale": "1-5", "weight": 0.4},
        {"name": "clarity", "description": "Explanation clarity", "scale": "1-5", "weight": 0.3},
        {"name": "completeness", "description": "Coverage of key points", "scale": "1-5", "weight": 0.3}
    ],
    "rubric": {
        "criteria": [],
        "scale_description": {
            "1": "Severely lacking",
            "2": "Below expectations",
            "3": "Meets basic requirements",
            "4": "Exceeds expectations",
            "5": "Outstanding quality"
        }
    },
    "judge_model": "gpt-4",
    "use_cot": True
})

🌐 REST API集成

# Evaluate response via REST API
curl -X POST http://localhost:8080/judge/evaluate \
  -H "Content-Type: application/json" \
  -d '{
    "response": "Paris is the capital of France",
    "criteria": [
      {
        "name": "accuracy",
        "description": "Factual accuracy",
        "scale": "1-5",
        "weight": 1.0
      }
    ],
    "rubric": {
      "criteria": [],
      "scale_description": {
        "1": "Wrong",
        "5": "Correct"
      }
    },
    "judge_model": "gpt-4o-mini"
  }'

# Python REST API client
import httpx
import asyncio

async def evaluate_via_rest():
    async with httpx.AsyncClient() as client:
        response = await client.post("http://localhost:8080/judge/evaluate", json={
            "response": "Technical explanation...",
            "criteria": [
                {"name": "quality", "description": "Overall quality", "scale": "1-5", "weight": 1.0}
            ],
            "rubric": {
                "criteria": [],
                "scale_description": {"1": "Poor", "5": "Excellent"}
            },
            "judge_model": "gpt-4o-mini"
        })
        result = response.json()
        return result

# Run evaluation
result = asyncio.run(evaluate_via_rest())
print(f"Overall score: {result['overall_score']}")

⚖️ 高级成对比较

# Head-to-head comparison with bias mitigation
comparison = await mcp_client.call_tool("judge.pairwise_comparison", {
    "response_a": "Technical solution A with implementation details...",
    "response_b": "Alternative solution B with different approach...",
    "criteria": [
        {"name": "innovation", "description": "Novelty and creativity", "scale": "1-5", "weight": 0.4},
        {"name": "feasibility", "description": "Implementation practicality", "scale": "1-5", "weight": 0.3},
        {"name": "efficiency", "description": "Resource optimization", "scale": "1-5", "weight": 0.3}
    ],
    "context": "Solutions for enterprise-scale data processing challenge",
    "position_bias_mitigation": True,
    "judge_model": "gpt-4-turbo"
})

📊 综合代理基准测试

# Full agent performance assessment
benchmark_result = await mcp_client.call_tool("agent.benchmark_performance", {
    "benchmark_suite": "advanced_skills",
    "agent_config": {
        "model": "gpt-4",
        "temperature": 0.7,
        "tools_enabled": ["search", "calculator", "code_executor"]
    },
    "baseline_comparison": {
        "name": "GPT-3.5 Baseline",
        "scores": {"accuracy": 0.75, "efficiency": 0.68, "reliability": 0.72}
    },
    "metrics_focus": ["accuracy", "efficiency", "reliability", "creativity"]
})

🔄 高级评估套件

# Create sophisticated evaluation pipeline
suite = await mcp_client.call_tool("workflow.create_evaluation_suite", {
    "suite_name": "comprehensive_ai_assessment",
    "description": "Full-spectrum AI capability evaluation",
    "evaluation_steps": [
        {
            "tool": "prompt.evaluate_clarity",
            "weight": 0.15,
            "parameters": {"target_model": "gpt-4", "domain_context": "technical"}
        },
        {
            "tool": "judge.evaluate_response",
            "weight": 0.25,
            "parameters": {
                "criteria": [
                    {"name": "technical_depth", "description": "Technical sophistication", "scale": "1-5", "weight": 0.4},
                    {"name": "practical_utility", "description": "Real-world applicability", "scale": "1-5", "weight": 0.6}
                ],
                "judge_model": "gpt-4"
            }
        },
        {
            "tool": "quality.evaluate_factuality",
            "weight": 0.20
        },
        {
            "tool": "quality.measure_coherence",
            "weight": 0.15
        },
        {
            "tool": "quality.assess_toxicity",
            "weight": 0.10
        },
        {
            "tool": "agent.analyze_reasoning",
            "weight": 0.15,
            "parameters": {"judge_model": "gpt-4-turbo"}
        }
    ],
    "success_thresholds": {
        "overall": 0.85,
        "quality.evaluate_factuality": 0.90,
        "quality.assess_toxicity": 0.95
    },
    "weights": {
        "accuracy": 0.4,
        "safety": 0.3,
        "utility": 0.3
    }
})

# Execute comprehensive evaluation
results = await mcp_client.call_tool("workflow.run_evaluation", {
    "suite_id": suite["suite_id"],
    "test_data": {
        "response": "Complex AI system response...",
        "context": "Enterprise deployment scenario...",
        "reasoning_trace": [...],
        "agent_trace": {...}
    },
    "parallel_execution": True,
    "max_concurrent": 5
})

🎛️ 高级配置

自定义模型配置

MCP评估服务器支持法官模型的完全定制，使您能够：

• 配置自定义API端点和部署
• 设置特定于提供商的参数和功能
• 创建特定于域的模型配置
• 使用自定义环境变量名

# Use custom model configuration
export MCP_EVAL_MODELS_CONFIG="./my-custom-models.yaml"
export DEFAULT_JUDGE_MODEL="my-custom-judge"

# Copy default config for customization
make copy-config                    # Copies to ./custom-config/
make show-config                    # Show current configuration status
make validate-config                # Validate configuration syntax

具有功能的模型配置

models:
  azure:
    my-enterprise-gpt4:
      provider: "azure"
      deployment_name: "my-gpt4-deployment"
      model_name: "gpt-4"
      api_base_env: "AZURE_OPENAI_ENDPOINT"
      api_key_env: "AZURE_OPENAI_API_KEY"
      api_version_env: "AZURE_OPENAI_API_VERSION"
      deployment_name_env: "AZURE_DEPLOYMENT_NAME"
      default_temperature: 0.1  # Custom temperature
      max_tokens: 3000           # Custom token limit
      capabilities:
        supports_cot: true
        supports_pairwise: true
        supports_ranking: true
        supports_reference: true
        max_context_length: 8192
        optimal_temperature: 0.1
        consistency_level: "very_high"
      metadata:
        purpose: "production_evaluation"
        cost_tier: "premium"

  ollama:
    my-local-llama:
      provider: "ollama"
      model_name: "llama3:70b"
      base_url_env: "OLLAMA_BASE_URL"
      default_temperature: 0.3
      max_tokens: 2000
      request_timeout: 120  # Longer timeout for large models

# Custom defaults
defaults:
  primary_judge: "my-enterprise-gpt4"
  fallback_judge: "my-local-llama"

# Custom recommendations
recommendations:
  production: ["my-enterprise-gpt4"]
  development: ["my-local-llama"]

高级评估量规

rubrics:
  technical_excellence:
    name: "Technical Excellence Assessment"
    criteria:
      - name: "code_quality"
        description: "Code structure, efficiency, and best practices"
        scale: "1-10"
        weight: 0.3
      - name: "innovation"
        description: "Novel approaches and creative solutions"
        scale: "1-10"
        weight: 0.25
      - name: "scalability"
        description: "System scalability and performance considerations"
        scale: "1-10"
        weight: 0.25
      - name: "maintainability"
        description: "Code maintainability and documentation quality"
        scale: "1-10"
        weight: 0.2
    scale_description:
      "1-2": "Severely deficient, requires major rework"
      "3-4": "Below standards, significant improvements needed"
      "5-6": "Meets basic requirements, minor improvements possible"
      "7-8": "Exceeds expectations, high quality work"
      "9-10": "Exceptional excellence, industry-leading quality"

多域基准

benchmarks:
  enterprise_readiness:
    name: "Enterprise Readiness Assessment"
    category: "production"
    tasks:
      - name: "security_analysis"
        description: "Security vulnerability assessment and mitigation"
        difficulty: "advanced"
        expected_tools: ["security_scanner", "vulnerability_analyzer", "mitigation_planner"]
        evaluation_metrics: ["threat_identification", "risk_assessment", "solution_quality"]
      - name: "performance_optimization"
        description: "System performance analysis and optimization"
        difficulty: "advanced"
        expected_tools: ["profiler", "optimizer", "benchmarker"]
        evaluation_metrics: ["performance_gain", "resource_efficiency", "scalability_impact"]

🔬 研究级特征

📊 统计分析

• 相关性分析：Pearson、Spearman、Cohen的Kappa用于协议测量
• 显著性检验：评价差异的统计验证
• 趋势分析：绩效轨迹分析和波动性评估
• 异常检测：评估结果中的异常识别
• 置信区间：评估分数的不确定性量化

🧪 实验能力

• 判断校准：系统的偏差检测和校正算法
• 评价标准的演变：机器学习驱动的量规优化
• 元评价：评价质量本身
• 人类对齐：根据专家的判断进行持续校准
• 交叉验证：评估可靠性的K倍验证

🎯 特定领域评估

• 技术内容：代码质量、架构评估、安全分析
• 创意写作：原创性、参与度、风格一致性评估
• 学术工作：研究质量、引文分析、论点强度
• 客户服务：乐于助人、礼貌、有效解决问题
• 教育内容：学习目标达成，教学清晰

🏗️ 生产架构

🔧 基础设施组件

• 多法官运行时：支持OpenAI、Azure OpenAI和基于规则的评估引擎
• 缓存层：Redis兼容的分布式缓存，自动失效
• 结果数据库：具有全面索引的SQLite/PPostgreSQL存储
• API网关：具有身份验证和速率限制的RESTful端点
• 监控系统：Prometheus指标与Grafana仪表板

📦 部署选项

• 容器部署：具有安全强化功能的生产就绪Docker/Podman容器
• Kubernetes支持：带有自动缩放和服务网格集成的Helm图表
• 云集成：AWS ECS、Azure容器实例、Google Cloud Run兼容性
• 边缘部署：用于边缘计算场景的轻量级容器
• 发展模式：具有调试功能的热重新加载开发服务器

🔒 安全与合规

• 企业安全：OAuth 2.0、JWT令牌、API密钥轮换
• 数据隐私：静态和传输中的加密、PII检测和过滤
• 审计日志：具有篡改检测功能的全面审计跟踪
• 合规就绪：支持SOC 2、GDPR、HIPAA合规框架
• 漏洞管理：持续的安全扫描和自动修补

🗺️ 工具生态系统图

🏆 MCP EVALUATION SERVER - 63 SPECIALIZED TOOLS 🏆
═══════════════════════════════════════════════════════════

📊 CORE EVALUATION SUITE (15 tools)
├── 🤖 Judge Tools (4) ────── LLM-as-a-judge evaluation
├── 📝 Prompt Tools (4) ───── Clarity, consistency, optimization
├── 🛠️ Agent Tools (4) ────── Performance, reasoning, benchmarking
└── 🔍 Quality Tools (3) ──── Factuality, coherence, toxicity

🔬 ADVANCED ASSESSMENT SUITE (39 tools)
├── 🔗 RAG Tools (8) ──────── Retrieval relevance, grounding, citations
├── ⚖️ Bias & Fairness (6) ── Demographic bias, intersectional analysis
├── 🛡️ Robustness (5) ──────── Adversarial testing, injection resistance
├── 🔒 Safety & Alignment (4) Harmful content, value alignment
├── 🌍 Multilingual (4) ────── Translation, cultural adaptation
├── ⚡ Performance (4) ──────── Latency, efficiency, scaling
└── 🔐 Privacy (8) ───────── PII detection, compliance, anonymization

🔧 SYSTEM MANAGEMENT (9 tools)
├── 🔄 Workflow Tools (3) ─── Evaluation suites, parallel execution
├── 📊 Calibration (2) ────── Judge agreement, rubric optimization
└── 🏥 Server Tools (4) ───── Health monitoring, system management

🎯 TOTAL: 63 TOOLS ACROSS 14 CATEGORIES 🎯

📋 完整的工具参考

裁判工具（4/63）

提示工具（4/63）

代理工具（4/63）

质量工具（3/63）

RAG工具（8/63）

偏见与公平工具（6/63）

稳健性工具（5/63）

安全和对齐工具（4/63）

多语言工具（4/63）

性能工具（4/63）

隐私工具（8/63）

工作流工具（3/63）

校准工具（2/63）

服务器工具（4/63）

💡 创新与研究整合

🧠 人工智能研究应用

• 模型比较研究：不同LLM架构的系统评估
• 快速工程研究：大规模即时有效性分析
• 代理人行为研究：全面的代理决策研究
• 偏差检测研究：跨模型的系统性偏差模式分析
• 评估方法：关于评价技术本身的元研究

🏢 企业应用程序

• 质量保证：生产系统中的自动化内容质量控制
• A/B测试：不同AI配置的系统比较
• 性能监控：对部署的人工智能系统进行持续评估
• 合规报告：自动生成评价合规性报告
• 成本优化：评估驱动的人工智能系统成本优化

🎓 教育应用

• 学生评估：学生人工智能项目的自动评估
• 课程开发：评估驱动的人工智能课程优化
• 研究培训：培训研究人员评估方法的工具
• 基准创建：制定新的评价基准
• 同行评审：用于学术工作的人工智能辅助同行评审系统

🚀 入门指南

🎯 部署选项快速参考

立即快速启动

选项1:MCP服务器（stdio）

# 1. Run MCP server (for Claude Desktop, etc.)
make dev                    # Shows connection info + starts server

# 2. Test basic functionality
make example               # Run evaluation example
make test-mcp             # Test MCP protocol

选项2:REST API服务器（FastAPI）

# 1. Run native REST API server
make serve-rest          # Starts on http://localhost:8080

# 2. Test REST API endpoints
make test-rest           # Test all REST endpoints

# 3. View interactive documentation
open http://localhost:8080/docs    # Swagger UI
open http://localhost:8080/redoc   # ReDoc

# 4. Get connection info
make rest-info           # Show complete REST API guide

选项3:HTTP网桥（基于HTTP的MCP）

# 1. Run MCP protocol over HTTP
make serve-http          # Starts on http://localhost:9000

# 2. Test HTTP endpoints
make test-http           # Test MCP JSON-RPC endpoints

# 3. Get connection info
make http-info           # Show complete HTTP bridge guide

选项4：Docker部署

# Build and deploy
make build && make run

集成示例

MCP客户端集成

# Basic MCP integration
from mcp import Client
client = Client("mcp-eval-server")

# Evaluate any AI output
result = await client.call_tool("judge.evaluate_response", {
    "response": "Your AI output here",
    "criteria": [{"name": "quality", "description": "Overall quality", "scale": "1-5", "weight": 1.0}],
    "rubric": {"criteria": [], "scale_description": {"1": "Poor", "5": "Excellent"}}
})

REST API集成

# Start REST API server
make serve-rest

# Check server health
curl http://localhost:8080/health

# List tool categories
curl http://localhost:8080/tools/categories

# Evaluate response directly via REST
curl -X POST http://localhost:8080/judge/evaluate \
  -H "Content-Type: application/json" \
  -d '{
    "response": "Paris is the capital of France.",
    "criteria": [
      {
        "name": "accuracy",
        "description": "Factual accuracy",
        "scale": "1-5",
        "weight": 1.0
      }
    ],
    "rubric": {
      "criteria": [],
      "scale_description": {"1": "Wrong", "5": "Correct"}
    },
    "judge_model": "rule-based"
  }'

HTTP网桥集成（MCP over HTTP）

# Start HTTP bridge server
make serve-http

# List available tools (JSON-RPC)
curl -X POST \
     -H "Content-Type: application/json" \
     -d '{"jsonrpc": "2.0", "id": 1, "method": "tools/list", "params": {}}' \
     http://localhost:9000/

# Evaluate response via HTTP bridge (JSON-RPC)
curl -X POST \
     -H "Content-Type: application/json" \
     -d '{
       "jsonrpc": "2.0",
       "id": 2,
       "method": "tools/call",
       "params": {
         "name": "judge.evaluate_response",
         "arguments": {
           "response": "Paris is the capital of France.",
           "criteria": [{"name": "accuracy", "description": "Factual accuracy", "scale": "1-5", "weight": 1.0}],
           "rubric": {"criteria": [], "scale_description": {"1": "Wrong", "5": "Correct"}},
           "judge_model": "rule-based"
         }
       }
     }' \
     http://localhost:9000/

Python REST API客户端集成

import httpx
import asyncio

async def evaluate_via_rest_api():
    """Example using native REST API endpoints."""
    async with httpx.AsyncClient() as client:
        base_url = "http://localhost:8080"

        # Check health
        health = await client.get(f"{base_url}/health")
        print(f"Server status: {health.json()['status']}")

        # List tool categories
        categories = await client.get(f"{base_url}/tools/categories")
        print(f"Available categories: {len(categories.json()['categories'])}")

        # Evaluate response using REST endpoint
        evaluation = await client.post(f"{base_url}/judge/evaluate", json={
            "response": "Your AI response here",
            "criteria": [
                {"name": "quality", "description": "Overall quality", "scale": "1-5", "weight": 1.0}
            ],
            "rubric": {
                "criteria": [],
                "scale_description": {"1": "Poor", "5": "Excellent"}
            },
            "judge_model": "rule-based"
        })
        result = evaluation.json()
        print(f"Evaluation score: {result['overall_score']}")

        # Check content toxicity
        toxicity = await client.post(f"{base_url}/quality/toxicity", json={
            "content": "This is a test message",
            "toxicity_categories": ["profanity", "hate_speech"],
            "sensitivity_level": "moderate",
            "judge_model": "rule-based"
        })
        result = toxicity.json()
        print(f"Toxicity detected: {result['toxicity_detected']}")

# Run evaluation
asyncio.run(evaluate_via_rest_api())

Python HTTP桥客户端集成

import httpx
import asyncio

async def evaluate_via_http_bridge():
    """Example using MCP over HTTP bridge."""
    async with httpx.AsyncClient() as client:
        base_url = "http://localhost:9000"

        # List tools via JSON-RPC
        tools_request = {
            "jsonrpc": "2.0",
            "id": 1,
            "method": "tools/list",
            "params": {}
        }

        response = await client.post(base_url, json=tools_request)
        result = response.json()
        tools = result.get("result", [])
        print(f"Available tools: {len(tools)}")

        # Evaluate response via JSON-RPC
        eval_request = {
            "jsonrpc": "2.0",
            "id": 2,
            "method": "tools/call",
            "params": {
                "name": "judge.evaluate_response",
                "arguments": {
                    "response": "Your AI response here",
                    "criteria": [{"name": "quality", "description": "Overall quality", "scale": "1-5", "weight": 1.0}],
                    "rubric": {"criteria": [], "scale_description": {"1": "Poor", "5": "Excellent"}},
                    "judge_model": "rule-based"
                }
            }
        }

        response = await client.post(base_url, json=eval_request)
        result = response.json()
        print(f"Evaluation result: {result}")

# Run evaluation
asyncio.run(evaluate_via_http_bridge())