multi-dim-eval-framework（多维度评估框架）

Multi-Dimensional Evaluation Framework Designer

A skill for designing custom multi-dimensional evaluation frameworks for AI systems. Walks the user from "I have a system to evaluate" to "I have a calibrated, group-organized scorecard with canonical/proxy duality and explicit failure modes."

The central premise: a single composite score destroys the information you need to debug *which* dimension actually drove the outcome. This skill produces frameworks that force the reader to look at multiple numbers, with rules for when each measurement is reliable.

Four-stage flow

• Stage 1 — Domain elicitation: what system, what evaluation question, what calibration cases
• Stage 2 — Taxonomy design: group structure + dimensions per group
• Stage 3 — Rubric: canonical/proxy split per dimension + failure modes
• Stage 4 — Judgment: group-wise scorecard interpretation (no composite)

After Stage 4, ask: *"Want to score additional cases or adjust the rubric?"* — this is the calibration loop.

When to use

Activate when the user:

• Wants to evaluate AI systems (agents, deliberations, RAG, multi-step reasoning) across multiple qualitatively-different dimensions
• Needs to compare instances with asymmetric data availability (some have canonical metrics, others have only narrative logs)
• Has noticed single-score benchmarks miss important variation between systems
• Says "tradeoffs" — and wants to make those tradeoffs explicit per dimension
• Wants a reusable scorecard format that survives infrastructure migrations

Don't activate when:

• The user wants a single comparable benchmark number — point them at HumanEval / MMLU / domain-specific benchmarks instead
• The system has a clear single quality metric (perplexity, accuracy on a labeled set)
• The user is asking how to design *one* metric, not a *framework* of metrics

Stage 1 — Domain elicitation

Goal: extract enough about the user's evaluation domain to design groups and dimensions.

Turn 1 — concrete instances, not abstract criteria. Ask:

"Give me 1-2 concrete instances of systems you want to evaluate (or have already evaluated). What's the question that comparison should answer? — e.g., 'is system V2 more grounded than V1?' / 'does adding a Critic agent reduce sycophancy?'"

This grounds the design in real comparisons rather than generic axes.

Turn 2 — calibration cases. Ask:

"Of the systems you've already run, which 2-3 do you have *strong intuitions* about — i.e., 'I expect X to score higher than Y because Z'? Those are your calibration cases."

If the user has no calibration cases yet, the framework can't be calibrated. Either:

• Run on at least 2 prior instances first, or
• Design the framework theoretically and acknowledge it's uncalibrated until run

Turn 3 — data availability. Ask:

"For each calibration case, what data do you have? — structured records (jsonl, database)? narrative logs (markdown, reports)? both? Same schema across cases or different?"

This determines canonical/proxy split for Stage 3.

Turn 4 — capability layers (optional). If the system is complex, ask:

"If you had to split the evaluation into 3 layers, what would they be? Examples: evidence-quality / process-dynamics / structural-form. Or: retrieval-quality / ranking-quality / adaptation-quality."

The user's natural splits become the groups. If the user can't articulate layers, default to a 3-group structure: (1) evidence/grounding, (2) process/dynamics, (3) structural/architecture. Or use the 4-family alternative shown in memory-bench-taxonomy.md.

By end of Stage 1 you should know:

• The system class being evaluated (multi-agent / single-LLM / RAG / tool-using / etc.)
• 2-3 calibration cases with expected ordinals
• Data availability map (which cases have canonical data, which need proxy)
• Group structure (typically 3 groups, may be 2 or 4)

Stage 2 — Taxonomy design

Author the group structure + dimensions per group.

Step 1: Surface the 12-axis MADEF reference to the user. Ask which axes feel relevant.

Don't force the user to use all 12 — most domains use 5-8 of the MADEF axes plus 0-3 domain-specific additions. The MADEF table at the bottom of madef-axes.md shows likely keep/modify/drop patterns for common domains (single-LLM reasoning, tool-using agents, RAG, multi-step coding).

Step 2: Show the memory-bench-designer's 4-family taxonomy as alternative shape.

This makes the point that group structure is domain-driven. memory-bench has 4 groups (capability families) because memory has those layers. Deliberation has 3 groups (evidence/process/structure) because deliberation has those layers. Don't blindly copy — let the user's domain shape it.

Step 3: Walk the design worksheet. Use axes-design-worksheet.md to fill in:

• Group names + what each layer asks
• 2-5 dimensions per group
• For each dimension: name + 1-line definition

Cap at 8-12 total dimensions. More than 12 is unmanageable; less than 4 isn't multi-dim.

Stage 3 — Rubric

For each dimension designed in Stage 2, fill in the operational rubric using canonical-vs-proxy-decision.md:

• Canonical measure (formula given full data)
• Fallback proxy (operationalization for partial data)
• Tie-break rule (partial credit cases)
• Flag conditions (when to attach ⚠)
• Refusal threshold (when proxy is too noisy to score)

A dimension without all five fields is not yet operational — it's a sketch.

Apply group-design-principles.md M1-M5 meta-principles:

• M1: ambiguous → report range, not point
• M2: population-count normalization required for cross-instance
• M3: stress conditions evaluated separately
• M4: framework must be falsifiable
• M5: calibration before claims

Stage 4 — Judgment

Apply the framework to the calibration cases the user named in Stage 1.

For each case, populate scorecard.md.tmpl with group-wise scores.

Critical: report group means separately, never a composite. A failing system with one group at 0.9 and another at 0.2 is not the same as a system with all groups at 0.55.

Verify ordinal predictions: do the calibration cases score in the predicted order? If not:

• Iterate the rubric and log the change in iteration_log.md (see group-design-principles.md M5)
• Or accept that the prediction was wrong and document why

The framework freezes (becomes versioned) when the calibration ordinals hold and at least 2-3 real adjustments have been logged.

Quick example

User: *"I have 4 multi-agent debate experiments. The 4th one added claims+verifications infra. I want to evaluate which experiment is doing the most rigorous deliberation."*

Stage 1 reveals:

• System class: multi-agent deliberation, 3-5 agents per experiment, 13-20 rounds each
• Calibration cases: V1/V2/V3 (legacy) and V4 (with claims infra)
• Data availability: legacy has narrative round logs only; V4 has full state jsonl
• Predicted ordinals: V2 > V1 (added Critic), V3 > V2 (more agents), V4 highest on grounding (has claims infra)

Stage 2 lands on the 12-axis MADEF taxonomy in madef-axes.md, with 3 groups (Grounding / Dynamics / Architecture).

Stage 3 fills in canonical/proxy for each axis. Most legacy experiments need proxy on A1, A3, B1, B2; V4 has canonical on all.

Stage 4 produces 4 scorecards. The ordinals confirm V4 is highest on Group A (Grounding) but the picture is more nuanced on Group B (V3 outscores V4 on dynamics due to more agents and a unique cross-agent finding). The framework surfaces *which* dimensions move with the architecture change, which is what the user needed.

Full walkthrough: examples/deliberation-system-eval.md.

How the skill behaves at each turn

• Don't dump all 12 axes at once. Surface them in groups, ask about relevance group-by-group.
• Don't start with the rubric (Stage 3) before the taxonomy is settled (Stage 2). Operational definitions before the design intent is wasted work.
• Do push back if the user wants a single composite. The pattern's whole point is to refuse that. Explain *why* (it hides which dimension failed) rather than just refusing.
• Do verify calibration ordinals before the user "trusts" the framework. If the framework can't reproduce the ordinals the user predicted, *something* is wrong (rubric, prediction, or scoring) — find which.

References

• references/group-design-principles.md — five design principles + five meta-principles, domain-agnostic
• references/canonical-vs-proxy-decision.md — decision tree for two-track measurement
• references/madef-axes.md — 12-axis instantiation for multi-agent deliberation (use as reference, adapt to your domain)
• references/memory-bench-taxonomy.md — 4-family/8-dimension instantiation for memory eval (alternative shape)

Templates

• templates/axes-design-worksheet.md — fill-in worksheet for designing your own axes
• templates/scorecard.md.tmpl — output format for group-wise scorecards

Examples

• examples/deliberation-system-eval.md — applying MADEF to 4 deliberation experiments
• examples/cross-domain-rag-eval.md — adapting the pattern to RAG evaluation

What this skill does NOT do

• It does not run benchmarks for you — it designs the framework you'll run
• It does not produce automated scoring — scoring is procedurally specified but human-in-the-loop for proxy work
• It does not collapse multi-dim into a single ranking number (refusal is the design)
• It does not validate that the dimensions you choose are *the right* dimensions for your domain — that's a calibration question, the framework only enforces self-consistency

License

MIT