Reality Validation
Compare system behavior against real-world expectations, domain knowledge, and practical constraints rather than just technical specifications to ensure software makes sense in actual usage contexts.
When to use me
Use this skill when:
- Building systems for complex real-world domains (finance, healthcare, logistics)
- Validating that technical solutions solve actual user problems
- Testing business logic against domain expert knowledge
- Ensuring data models match real-world entities and relationships
- Verifying that algorithms produce realistic and practical results
- Checking that user interfaces reflect real user workflows
- Validating reports and analytics against business reality
- Preparing for production deployment in mission-critical domains
- Debugging issues where technical correctness ≠ real-world correctness
What I do
1. Domain Understanding & Context Building
- Research real-world context:
- Study domain terminology, concepts, and relationships - Understand business processes and workflows - Learn about user roles, responsibilities, and goals - Identify constraints, regulations, and industry standards - Map real-world entities to system representations
- Gather domain knowledge sources:
- Subject matter experts (SMEs) and stakeholders - Industry documentation and standards - Historical data and case studies - User observations and interviews - Competitor and market analysis - Regulatory and compliance requirements
2. Reality Gap Analysis
- Compare system behavior to real-world expectations:
- Data validation: Does system data match real-world data patterns? - Process validation: Do workflows match actual business processes? - Calculation validation: Do calculations produce realistic results? - Timing validation: Do time-based behaviors match reality? - Scale validation: Does system scale match real-world volumes? - Edge case validation: Do edge cases reflect real-world scenarios?
- Identify mismatches and anomalies:
- Technical correctness vs practical usefulness - Specification compliance vs real-world applicability - Algorithmic precision vs business acceptability - System limitations vs real-world requirements - Assumed user behavior vs actual user behavior
3. Reality-Based Testing
- Test with real-world data:
- Production data samples (sanitized) - Historical transaction data - Real user scenarios and cases - Actual business events and conditions - Real-time data feeds and updates
- Validate against domain expertise:
- Expert review of system outputs - SME validation of calculations and logic - User acceptance testing with real users - Business stakeholder verification - Regulatory compliance validation
- Execute context-aware validation:
- Seasonal variation testing - Peak load and stress scenario testing - Geographic and regional variation testing - Market condition simulation - Regulatory change impact testing
4. Practical Correctness Assessment
- Evaluate beyond technical correctness:
- Are results practically useful? - Do outputs make business sense? - Are workflows efficient in practice? - Is data quality sufficient for decision-making? - Does system support real user goals?
- Assess real-world impact:
- Business value delivered vs expected - User satisfaction and adoption - Operational efficiency improvements - Risk reduction and compliance - Competitive advantage gained
Reality Validation Areas
Data & Model Validation:
- Data patterns: Do system data distributions match real-world distributions?
- Entity relationships: Do modeled relationships reflect real relationships?
- Data quality: Is data accurate, complete, and timely enough for real use?
- Temporal patterns: Do time-based behaviors match real-world timing?
- Geographic patterns: Do location-based behaviors make geographic sense?
Process & Workflow Validation:
- Business processes: Do automated processes match manual processes?
- User workflows: Do system workflows support actual user tasks?
- Decision points: Do system decisions reflect real decision criteria?
- Approval flows: Do approval processes match organizational reality?
- Exception handling: Do exceptions reflect real exceptional situations?
Calculation & Logic Validation:
- Business calculations: Do calculations produce business-valid results?
- Pricing logic: Does pricing match market reality and business strategy?
- Risk assessments: Do risk calculations reflect actual risk factors?
- Forecasting: Do forecasts align with historical patterns and expert expectations?
- Optimization: Do optimization results provide practical improvements?
User Experience Validation:
- User mental models: Does UI match user understanding of the domain?
- Workflow efficiency: Do interfaces support efficient real work?
- Information presentation: Is information presented for practical use?
- Decision support: Does system help users make better real decisions?
- Learning curve: Can users apply domain knowledge to system use?
Examples
# Domain-specific reality validation
npm run reality:validate -- --domain healthcare --system "patient-scheduling"
npm run reality:validate -- --domain finance --system "portfolio-management"
npm run reality:validate -- --domain ecommerce --system "inventory-management"
# Data reality validation
npm run reality:validate:data -- --dataset production-samples
npm run reality:validate:data -- --compare historical-actual
npm run reality:validate:data -- --expert-review business-analyst
# Process reality validation
npm run reality:validate:process -- --workflow "order-fulfillment"
npm run reality:validate:process -- --observe real-users
npm run reality:validate:process -- --compare manual-process
# Calculation reality validation
npm run reality:validate:calculations -- --expert finance-director
npm run reality:validate:calculations -- --compare industry-benchmarks
npm run reality:validate:calculations -- --test real-scenarios
# Integration with other testing
npm run reality:validate:with -- --test-type usability --focus "workflow-reality"
npm run reality:validate:with -- --test-type performance --focus "real-world-load"
npm run reality:validate:with -- --test-type security --focus "regulatory-compliance"
# Context-aware validation
npm run reality:validate:context -- --season holiday-peak
npm run reality:validate:context -- --market-condition recession
npm run reality:validate:context -- --geographic-region europe
npm run reality:validate:context -- --regulatory-environment gdprOutput format
Reality Validation Report
──────────────────────────────
Domain: Healthcare - Patient Appointment Scheduling
System: MedSchedule v2.1
Validation Method: Expert review + historical data analysis
Duration: 2 days
Data Reality Assessment:
1. Patient Appointment Duration Modeling:
System Assumption: All appointments are 30 minutes
Reality Check: ❌ MISMATCH FOUND
- Historical data shows: 15% are 15min, 60% are 30min, 20% are 45min, 5% are 60min
- Specialist appointments vary by department
- Follow-up visits often shorter than initial consultations
Impact: Schedule inefficiencies, doctor idle time, patient waiting
Recommendation: Implement variable appointment durations by type
2. Doctor Availability Patterns:
System Assumption: Doctors available 9-5, Monday-Friday
Reality Check: ⚠️ PARTIAL MATCH
- Reality: Varies by department (ER: 24/7, Surgery: 7-7, Clinic: 8-6)
- Reality: Weekend and holiday coverage differs
- Reality: On-call schedules not represented
Impact: Appointment booking outside real availability
Recommendation: Department-specific availability modeling
3. Patient No-Show Rate:
System Assumption: Constant 10% no-show rate
Reality Check: ❌ MISMATCH FOUND
- Reality: Varies by department (5-25%)
- Reality: Higher for Medicaid patients (18%)
- Reality: Lower for established patients (7%)
- Reality: Time-of-day and day-of-week patterns exist
Impact: Schedule optimization ineffective, resource waste
Recommendation: Dynamic no-show prediction model
Process Reality Assessment:
1. Appointment Rescheduling Workflow:
System Design: Patient calls → receptionist checks availability → confirms
Reality Check: ✅ GOOD MATCH
- Matches observed clinic workflows
- Receptionist role accurately modeled
- Phone channel primary for rescheduling
Impact: System supports actual process well
2. Emergency Appointment Handling:
System Design: Treated as regular appointment with priority flag
Reality Check: ❌ MISMATCH FOUND
- Reality: ER has separate booking system
- Reality: Urgent care uses walk-in model
- Reality: True emergencies go to ER, not scheduled
Impact: System doesn't support real emergency handling
Recommendation: Separate emergency/urgent care workflow
3. Referral Process Integration:
System Design: Doctor creates referral in system
Reality Check: ⚠️ PARTIAL MATCH
- Reality: Paper referrals still common (40% of cases)
- Reality: Insurance pre-authorization often required
- Reality: Specialist availability check happens separately
Impact: Digital workflow incomplete, manual workarounds needed
Recommendation: Integrate paper referral scanning and insurance checks
Calculation Reality Assessment:
1. Schedule Optimization Algorithm:
System Output: Maximizes doctor utilization
Reality Check: ❌ PRACTICALLY INVALID
- Doctors report algorithm creates "inhuman" schedules
- No buffer between patients for charting
- Doesn't account for doctor preferences or energy patterns
- Patient travel time between buildings not considered
Impact: Algorithmically optimal but practically unusable
Recommendation: Incorporate human factors and practical constraints
2. Wait Time Calculation:
System Calculation: Based on schedule gaps
Reality Check: ⚠️ INACCURATE
- Reality: Actual wait times 2-3x calculated times
- Reasons: Doctor running late, emergency interruptions, complex cases
- Patients care about actual wait, not scheduled wait
Impact: Patient dissatisfaction due to inaccurate expectations
Recommendation: Implement actual wait time tracking and prediction
Expert Validation Results:
- Clinical Director: "System doesn't understand real clinic workflow"
- Head Nurse: "Missing critical nurse coordination aspects"
- IT Director: "Technically sound but practically limited"
- Patients: "Easy to use but schedules often inaccurate"
Business Impact Assessment:
- Schedule Efficiency: Current 68%, Potential 85% with improvements
- Patient Satisfaction: Current 3.2/5, Potential 4.5/5 with fixes
- Doctor Utilization: Current 75%, Potential 82% with better modeling
- Administrative Burden: Current high, Potential medium with improvements
Reality Gap Summary:
- Major Gaps: 4 (critical impact on system usefulness)
- Moderate Gaps: 6 (significant impact on efficiency)
- Minor Gaps: 3 (noticeable but manageable)
- Well Aligned: 5 aspects match reality well
Recommendations by Priority:
1. High: Implement variable appointment durations
2. High: Add emergency/urgent care workflow separation
3. Medium: Develop dynamic no-show prediction
4. Medium: Incorporate human factors in scheduling
5. Low: Improve wait time calculation accuracy
Validation Confidence:
- Data Alignment: Low (significant mismatches found)
- Process Alignment: Medium (some mismatches, mostly aligned)
- Calculation Alignment: Low (theoretically sound, practically weak)
- Overall Reality Fit: Medium-Low (needs substantial improvements)
Next Steps:
1. Schedule workshops with clinical staff to redesign key workflows
2. Implement highest priority reality fixes before next release
3. Establish ongoing reality validation process with domain experts
4. Track reality alignment metrics over timeNotes
- Reality validation requires deep domain understanding, not just technical skill
- Involve domain experts early and often
- Real-world data often contradicts assumptions and simplifications
- Technical elegance doesn't guarantee practical usefulness
- Balance theoretical correctness with practical constraints
- Different domains have different reality validation needs
- Reality changes over time - validation should be ongoing
- Document reality gaps and their business impacts clearly
- Use reality validation to bridge technical and business perspectives
- The most successful systems align closely with reality, not just specifications
- Reality validation often reveals that the real problem differs from the technical problem being solved