Increment Quality Judge v2.0
LLM-as-Judge Pattern Implementation
AI-powered quality assessment using the LLM-as-Judge pattern - an established AI/ML evaluation technique where an LLM evaluates outputs with chain-of-thought reasoning, BMAD-pattern risk scoring, and formal quality gate decisions (PASS/CONCERNS/FAIL).
LLM-as-Judge: What It Is
LLM-as-Judge (LaaJ) is a recognized pattern in AI/ML evaluation where a large language model assesses quality using structured reasoning.
┌─────────────────────────────────────────────────────────────┐
│ LLM-as-Judge Pattern │
├─────────────────────────────────────────────────────────────┤
│ Input: spec.md, plan.md, tasks.md │
│ │
│ Process: │
│ ┌─────────────────────────────────────────────────────┐ │
│ │ <thinking> │ │
│ │ 1. Read and understand the specification │ │
│ │ 2. Evaluate against 7 quality dimensions │ │
│ │ 3. Identify risks (P×I scoring) │ │
│ │ 4. Form evidence-based verdict │ │
│ │ </thinking> │ │
│ └─────────────────────────────────────────────────────┘ │
│ │
│ Output: Structured verdict with: │
│ • Dimension scores (0-100) │
│ • Risk assessment (CRITICAL/HIGH/MEDIUM/LOW) │
│ • Quality gate decision (PASS/CONCERNS/FAIL) │
│ • Actionable recommendations │
└─────────────────────────────────────────────────────────────┘Why LLM-as-Judge works:
- Consistency: Uniform evaluation criteria without human fatigue
- Reasoning: Chain-of-thought explains WHY something is an issue
- Scalability: Evaluates in seconds vs hours of manual review
- Industry standard: Used by OpenAI, Anthropic, Google for AI evals
References:
- "Judging LLM-as-a-Judge" (NeurIPS 2023)
- LMSYS Chatbot Arena evaluation methodology
- AlpacaEval, MT-Bench frameworks
IMPORTANT: This is a SKILL (Not an Agent)
DO NOT try to spawn this as an agent via Task tool.
This is a skill that auto-activates when you discuss quality assessment. To run quality assessment:
# Use the CLI command directly
specweave qa 0001 --pre
# Or use the slash command
/sw:qa 0001The skill provides guidance and documentation. The CLI handles execution.
Why no agent? Having both a skill and agent with the same name (increment-quality-judge-v2) caused Claude to incorrectly construct agent type names. The skill-only approach eliminates this confusion.
What's New in v2.0
- Risk Assessment Dimension - Probability × Impact scoring (0-10 scale, BMAD pattern)
- Quality Gate Decisions - Formal PASS/CONCERNS/FAIL with thresholds
- NFR Checking - Non-functional requirements (performance, security, scalability)
- Enhanced Output - Blockers, concerns, recommendations with actionable mitigations
- 7 Dimensions - Added "Risk" to the existing 6 dimensions
Purpose
Provide comprehensive quality assessment that goes beyond structural validation to evaluate:
- ✅ Specification quality (6 dimensions)
- ✅ Risk levels (BMAD P×I scoring) - NEW!
- ✅ Quality gate readiness (PASS/CONCERNS/FAIL) - NEW!
When to Use
Auto-activates for:
/qa {increment-id}command/qa {increment-id} --pre(pre-implementation check)/qa {increment-id} --gate(quality gate check)- Natural language: "assess quality of increment 0001"
Keywords:
- validate quality, quality check, assess spec
- evaluate increment, spec review, quality score
- risk assessment, qa check, quality gate
- PASS/CONCERNS/FAIL
Evaluation Dimensions (7 total, was 6)
dimensions:
clarity:
weight: 0.18 # was 0.20
criteria:
- "Is the problem statement clear?"
- "Are objectives well-defined?"
- "Is terminology consistent?"
testability:
weight: 0.22 # was 0.25
criteria:
- "Are acceptance criteria testable?"
- "Can success be measured objectively?"
- "Are edge cases identifiable?"
completeness:
weight: 0.18 # was 0.20
criteria:
- "Are all requirements addressed?"
- "Is error handling specified?"
- "Are non-functional requirements included?"
feasibility:
weight: 0.13 # was 0.15
criteria:
- "Is the architecture scalable?"
- "Are technical constraints realistic?"
- "Is timeline achievable?"
maintainability:
weight: 0.09 # was 0.10
criteria:
- "Is design modular?"
- "Are extension points identified?"
- "Is technical debt addressed?"
edge_cases:
weight: 0.09 # was 0.10
criteria:
- "Are failure scenarios covered?"
- "Are performance limits specified?"
- "Are security considerations included?"
# NEW: Risk Assessment (BMAD pattern)
risk:
weight: 0.11 # NEW!
criteria:
- "Are security risks identified and mitigated?"
- "Are technical risks (scalability, performance) addressed?"
- "Are implementation risks (complexity, dependencies) managed?"
- "Are operational risks (monitoring, support) considered?"Risk Assessment (BMAD Pattern) - NEW!
Risk Scoring Formula
Risk Score = Probability × Impact
Probability (0.0-1.0):
- 0.0-0.3: Low (unlikely to occur)
- 0.4-0.6: Medium (may occur)
- 0.7-1.0: High (likely to occur)
Impact (1-10):
- 1-3: Minor (cosmetic, no user impact)
- 4-6: Moderate (some impact, workaround exists)
- 7-9: Major (significant impact, no workaround)
- 10: Critical (system failure, data loss, security breach)
Final Score (0.0-10.0):
- 9.0-10.0: CRITICAL risk (FAIL quality gate)
- 6.0-8.9: HIGH risk (CONCERNS quality gate)
- 3.0-5.9: MEDIUM risk (PASS with monitoring)
- 0.0-2.9: LOW risk (PASS)Risk Categories
- Security Risks
- OWASP Top 10 vulnerabilities - Data exposure, authentication, authorization - Cryptographic failures
- Technical Risks
- Architecture complexity, scalability bottlenecks - Performance issues, technical debt
- Implementation Risks
- Tight timeline, external dependencies - Technical complexity
- Operational Risks
- Lack of monitoring, difficult to maintain - Poor documentation
Risk Assessment Prompt
You are evaluating SOFTWARE RISKS for an increment using BMAD's Probability × Impact scoring.
Read increment files:
- .specweave/increments/{id}/spec.md
- .specweave/increments/{id}/plan.md
For EACH risk you identify:
1. **Calculate PROBABILITY** (0.0-1.0)
- Based on spec clarity, past experience, complexity
- Low: 0.2, Medium: 0.5, High: 0.8
2. **Calculate IMPACT** (1-10)
- 10 = Critical (security breach, data loss, system failure)
- 7-9 = Major (significant user impact, no workaround)
- 4-6 = Moderate (some impact, workaround exists)
- 1-3 = Minor (cosmetic, no user impact)
3. **Calculate RISK SCORE** = Probability × Impact
4. **Provide MITIGATION** strategy
5. **Link to ACCEPTANCE CRITERIA** (if applicable)
Output format (JSON):
{
"risks": [
{
"id": "RISK-001",
"category": "security",
"title": "Password storage not specified",
"description": "Spec doesn't mention password hashing algorithm",
"probability": 0.9,
"impact": 10,
"score": 9.0,
"severity": "CRITICAL",
"mitigation": "Use bcrypt or Argon2, never plain text",
"location": "spec.md, Authentication section",
"acceptance_criteria": "AC-US1-01"
}
],
"overall_risk_score": 7.5,
"dimension_score": 0.35
}Quality Gate Decisions - NEW!
Decision Logic
enum QualityGateDecision {
PASS = "PASS", // Ready for production
CONCERNS = "CONCERNS", // Issues found, should address
FAIL = "FAIL" // Blockers, must fix
}
Thresholds (BMAD pattern):
FAIL if any:
- Risk score ≥ 9.0 (CRITICAL)
- Test coverage < 60%
- Spec quality < 50
- Critical security vulnerabilities ≥ 1
CONCERNS if any:
- Risk score 6.0-8.9 (HIGH)
- Test coverage < 80%
- Spec quality < 70
- High security vulnerabilities ≥ 1
PASS otherwiseOutput Example
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
QA ASSESSMENT: Increment 0008-user-authentication
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Overall Score: 82/100 (GOOD) ✓
Dimension Scores:
Clarity: 90/100 ✓✓
Testability: 75/100 ⚠️
Completeness: 88/100 ✓
Feasibility: 85/100 ✓
Maintainability: 80/100 ✓
Edge Cases: 70/100 ⚠️
Risk Assessment: 65/100 ⚠️ (7.2/10 risk score)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
RISKS IDENTIFIED (3)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔴 RISK-001: CRITICAL (9.0/10)
Category: Security
Title: Password storage implementation
Description: Spec doesn't specify password hashing
Probability: 0.9 (High) × Impact: 10 (Critical)
Location: spec.md, Authentication section
Mitigation: Use bcrypt/Argon2, never plain text
AC: AC-US1-01
🟡 RISK-002: HIGH (6.0/10)
Category: Security
Title: Rate limiting not specified
Description: No brute-force protection mentioned
Probability: 0.6 (Medium) × Impact: 10 (Critical)
Location: spec.md, Security section
Mitigation: Add 5 failed attempts → 15 min lockout
AC: AC-US1-03
🟢 RISK-003: LOW (2.4/10)
Category: Technical
Title: Session storage scalability
Description: Plan uses in-memory sessions
Probability: 0.4 (Medium) × Impact: 6 (Moderate)
Location: plan.md, Architecture section
Mitigation: Use Redis for session store
Overall Risk Score: 7.2/10 (MEDIUM-HIGH)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
QUALITY GATE DECISION
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🟡 CONCERNS (Not Ready for Production)
Blockers (MUST FIX):
1. 🔴 CRITICAL RISK: Password storage (Risk ≥9)
→ Add task: "Implement bcrypt password hashing"
Concerns (SHOULD FIX):
2. 🟡 HIGH RISK: Rate limiting not specified (Risk ≥6)
→ Update spec.md: Add rate limiting section
→ Add E2E test for rate limiting
3. ⚠️ Testability: 75/100 (target: 80+)
→ Make acceptance criteria more measurable
Recommendations (NICE TO FIX):
4. Edge cases: 70/100
→ Add error handling scenarios
5. Session scalability
→ Consider Redis for session store
Decision: Address 1 blocker before proceeding
Would you like to:
[E] Export blockers to tasks.md
[U] Update spec.md with fixes (experimental)
[C] Continue without changesWorkflow Integration
Quick Mode (Default)
User: /sw:qa 0001
Step 1: Rule-based validation (120 checks) - FREE, FAST
├── If FAILED → Stop, show errors
└── If PASSED → Continue
Step 2: AI Quality Assessment (Quick)
├── Spec quality (6 dimensions)
├── Risk assessment (BMAD P×I)
└── Quality gate decision (PASS/CONCERNS/FAIL)
Output: Enhanced report with risks and gate decisionPre-Implementation Mode
User: /sw:qa 0001 --pre
Checks:
✅ Spec quality (clarity, testability, completeness)
✅ Risk assessment (identify issues early)
✅ Architecture review (plan.md soundness)
✅ Test strategy (test plan in tasks.md)
Gate decision before implementation startsQuality Gate Mode
User: /sw:qa 0001 --gate
Comprehensive checks:
✅ All pre-implementation checks
✅ Test coverage (AC-ID coverage, gaps)
✅ E2E test coverage
✅ Documentation completeness
Final gate decision before closing incrementEnhanced Scoring Algorithm
Step 1: Dimension Evaluation (7 dimensions)
For each dimension (including NEW risk dimension), use Chain-of-Thought prompting:
<thinking>
1. Read spec.md thoroughly
2. For risk dimension specifically:
- Identify all risks (security, technical, implementation, operational)
- For each risk: calculate P, I, Score
- Group by category
- Calculate overall risk score
3. For other dimensions: evaluate criteria as before
4. Score 0.00-1.00
5. Identify issues
6. Provide suggestions
</thinking>
Score: 0.XXStep 2: Weighted Overall Score (NEW weights)
overall_score =
(clarity * 0.18) +
(testability * 0.22) +
(completeness * 0.18) +
(feasibility * 0.13) +
(maintainability * 0.09) +
(edge_cases * 0.09) +
(risk * 0.11) // NEW!Step 3: Quality Gate Decision
gate_decision = decide({
spec_quality: overall_score,
risk_score: risk_assessment.overall_risk_score,
test_coverage: test_coverage.percentage, // if available
security_audit: security_audit // if available
})Token Usage
Estimated per increment (Quick mode):
- Small spec (<100 lines): 2,500 tokens ($0.025)
- Medium spec (100-250 lines): 3,500 tokens ($0.035)
- Large spec (>250 lines): 5,000 tokens ($0.050)
Cost increase from v1.0: +25% (added risk assessment dimension)
Optimization:
- Only evaluate spec.md + plan.md for risks
- Cache risk patterns for 5 min
- Skip risk assessment if spec < 50 lines (too small to assess)
Configuration
{
"qa": {
"qualityGateThresholds": {
"fail": {
"riskScore": 9.0,
"testCoverage": 60,
"specQuality": 50,
"criticalVulnerabilities": 1
},
"concerns": {
"riskScore": 6.0,
"testCoverage": 80,
"specQuality": 70,
"highVulnerabilities": 1
}
},
"dimensions": {
"risk": {
"enabled": true,
"weight": 0.11
}
}
}
}Migration from v1.0
v1.0 (6 dimensions):
- Clarity, Testability, Completeness, Feasibility, Maintainability, Edge Cases
v2.0 (7 dimensions, NEW: Risk):
- All v1.0 dimensions + Risk Assessment
- Weights adjusted to accommodate new dimension
- Quality gate decisions added
- BMAD risk scoring added
Backward Compatibility:
- v1.0 skills still work (auto-upgrade to v2.0 if risk assessment enabled)
- Existing scores rescaled to new weights automatically
- Can disable risk assessment in config to revert to v1.0 behavior
Best Practices
- Run early and often: Use
--premode before implementation - Fix blockers immediately: Don't proceed if FAIL
- Address concerns before release: CONCERNS = should fix
- Use risk scores to prioritize: Fix CRITICAL risks first
- Export to tasks.md: Convert blockers/concerns to actionable tasks
Limitations
What quality-judge v2.0 CAN'T do:
- ❌ Understand domain-specific compliance (HIPAA, PCI-DSS)
- ❌ Verify technical feasibility with actual codebase
- ❌ Replace human expertise and security audits
- ❌ Predict actual probability without historical data
What quality-judge v2.0 CAN do:
- ✅ Catch vague or ambiguous language
- ✅ Identify missing security considerations (OWASP-based)
- ✅ Spot untestable acceptance criteria
- ✅ Suggest industry best practices
- ✅ Flag missing edge cases
- ✅ Assess risks systematically (BMAD pattern) - NEW!
- ✅ Provide formal quality gate decisions - NEW!
Summary
increment-quality-judge v2.0 adds comprehensive risk assessment and quality gate decisions:
✅ Risk assessment (BMAD P×I scoring, 0-10 scale) ✅ Quality gate decisions (PASS/CONCERNS/FAIL with thresholds) ✅ 7 dimensions (added "Risk" to existing 6) ✅ NFR checking (performance, security, scalability) ✅ Enhanced output (blockers, concerns, recommendations) ✅ Chain-of-thought (LLM-as-Judge 2025 best practices) ✅ Backward compatible (can disable risk assessment)
Use it when: You want comprehensive quality assessment with risk scoring and formal gate decisions before implementation or release.
Skip it when: Quick iteration, tight token budget, or simple features where rule-based validation suffices.
Version: 2.0.0 Related: /sw:qa command, QAOrchestrator agent