interview-system-designer（面试系统设计师）

Interview System Designer

The agent designs role-specific interview loops, generates competency-based question banks with scoring rubrics, and detects interviewer bias through statistical calibration analysis.

Quick Start

# Design a complete interview loop for a senior software engineer role
python loop_designer.py --role "Senior Software Engineer" --level senior --team platform --output loops/

# Generate a question bank for a product manager position
python question_bank_generator.py --role "Product Manager" --level senior --competencies leadership,strategy,analytics --output questions/

# Analyze interview calibration across candidates and interviewers
python hiring_calibrator.py --input interview_data.json --output calibration_report.json --analysis-type full

Core Workflows

Workflow 1: Design an Interview Loop

1. Define role requirements (title, level, team, 3-5 critical competencies)
2. Run loop_designer.py with role parameters to generate rounds, time allocations, and scorecards
3. Review generated loop for competency coverage -- every required competency maps to at least one round
4. Customize interviewer skill requirements per round
5. Validation checkpoint: 100% competency coverage; no round exceeds 90 minutes; total loop under 6 hours

python loop_designer.py --role "Staff Data Scientist" --level staff \
  --competencies ml,statistics,leadership --format json --output loops/ds-staff.json

Workflow 2: Generate a Question Bank

1. Identify target role and experience level
2. Select competency areas and question types (technical, behavioral, situational)
3. Run question_bank_generator.py to produce questions with scoring rubrics
4. Review for duplicate or overlapping questions across competency areas
5. Validation checkpoint: <15% duplicate rate; each competency has 3+ questions; calibration examples (poor/good/great) present for every question

python question_bank_generator.py --role "Frontend Engineer" \
  --competencies react,typescript,system-design --num-questions 30

Workflow 3: Calibrate Hiring Bar

1. Collect interview results data (minimum 10 records for statistical significance)
2. Run hiring_calibrator.py with comprehensive analysis
3. Review interviewer deviation metrics -- flag anyone >0.5 standard deviations from team mean
4. Generate coaching recommendations for flagged interviewers
5. Validation checkpoint: Bias detection precision >80%; score distribution follows target (20/40/30/10 split)

python hiring_calibrator.py --input q1_interviews.json \
  --analysis-type comprehensive --trend-analysis --period quarterly

Interview Loop Templates

Software Engineering Loops

Senior Software Engineer Example:

1. Technical Phone Screen (45min) -- Advanced algorithms, optimization
2. System Design (60min) -- Scalability, trade-offs, architectural decisions
3. Coding Excellence (60min) -- Code quality, testing strategies, refactoring
4. Technical Leadership (45min) -- Mentoring, technical decisions, cross-team collaboration
5. Behavioral & Culture (30min) -- Leadership examples, conflict resolution

Sample Questions by Level

Junior: "Implement a function to find the second largest element in an array" Senior: "Design a real-time chat system supporting 1M concurrent users" Staff+: "How would you evaluate and introduce a new programming language to the organization?"

Behavioral (STAR Method):

• "Tell me about a time you had to influence a decision without formal authority"
• "Walk me through a time when you had to make a decision with incomplete information"

Scoring Rubric

4-Point Scale

Calibration Benchmarks

• Target distribution: 20% (4s), 40% (3s), 30% (2s), 10% (1s)
• Interviewer consistency: <0.5 std dev from team average
• Pass rate: 15-25% for most roles
• New hire correlation: >0.6 between interview scores and 6-month performance

Anti-Patterns

• Unstandardized loops -- different question sets per candidate prevent fair comparison; always use structured guides
• Halo effect scoring -- one strong answer inflates all dimensions; score each competency independently before debrief
• Similarity bias -- favoring candidates with similar backgrounds; require diverse panels and rotate assignments
• Skipping calibration -- interviewers drift over time without regular calibration sessions (monthly minimum)
• Over-indexing on algorithms -- testing LeetCode for a staff role that requires architecture and leadership; match round focus to actual job requirements
• No debrief structure -- unstructured debriefs lead to anchoring on the loudest voice; require independent score submission before group discussion

Troubleshooting

Success Criteria

• Interview loop coverage: Every generated loop maps 100% of required competencies to at least one round with a dedicated scoring dimension.
• Question bank diversity: Generated banks contain no more than 15% duplicate or near-duplicate questions across competency areas.
• Calibration detection accuracy: Bias detection flags interviewer score deviation greater than 0.5 standard deviations from the team mean with at least 80% precision.
• Time-to-design reduction: Designing a complete interview loop (rounds, scorecards, question sets) takes under 10 minutes compared to the typical 2-4 hours of manual design.
• Rubric consistency: Generated scoring rubrics achieve inter-rater reliability (Cohen's kappa) of 0.7 or higher when tested with calibration panels.
• Candidate experience alignment: Loops designed with this tool target a candidate experience satisfaction score of 4.0/5.0 or above.
• Hiring quality signal: Organizations using the calibrator report a correlation of 0.6 or higher between interview scores and 6-month performance reviews.

Scope & Limitations

This skill covers:

• Designing end-to-end interview loops for engineering, product, design, and data roles across all seniority levels (junior through principal)
• Generating competency-based question banks with structured scoring rubrics and calibration examples
• Detecting statistical bias and calibration drift across interviewers and time periods
• Producing scorecard templates, debrief guides, and interviewer assignment recommendations

This skill does NOT cover:

• Applicant tracking system (ATS) integration, job posting, or candidate sourcing pipeline management — see hr-operations/talent-acquisition
• Compensation benchmarking, offer negotiation strategy, or total rewards analysis — see hr-operations/hr-business-partner
• Workforce planning, headcount modeling, or organizational design — see hr-operations/people-analytics
• Post-hire onboarding program design or new-hire ramp-up tracking — see engineering/codebase-onboarding

Integration Points

Tool Reference

loop_designer.py

Purpose: Generates calibrated interview loops tailored to specific roles, levels, and teams. Produces complete loops with rounds, focus areas, time allocation, interviewer skill requirements, and scorecard templates.

Usage:

python loop_designer.py --role "Senior Software Engineer" --level senior --team platform --output loops/

Flags/Parameters:

Example:

python loop_designer.py --role "Staff Data Scientist" --level staff --competencies ml,statistics,leadership --format json --output loops/ds-staff.json

Output Formats:

• JSON: Structured loop definition with rounds array, competency mappings, time allocations, and scorecard templates suitable for programmatic consumption.
• Text: Human-readable interview guide with formatted round descriptions, interviewer requirements, and evaluation criteria.
• Both (default): Writes both JSON and text outputs to the specified directory.

question_bank_generator.py

Purpose: Generates comprehensive, competency-based interview questions with detailed scoring criteria, follow-up probes, and calibration examples organized by competency area.

Usage:

python question_bank_generator.py --role "Frontend Engineer" --competencies react,typescript,system-design --output questions/

Flags/Parameters:

Example:

python question_bank_generator.py --role "Product Manager" --level mid --question-types behavioral,situational --num-questions 30 --format text

Output Formats:

• JSON: Array of question objects each containing the question text, competency area, difficulty level, scoring rubric (1-4 scale), follow-up probes, and calibration examples (poor/good/great answers).
• Text: Formatted question bank grouped by competency with inline scoring guidance and example answers for interviewer reference.
• Both (default): Writes both JSON and text outputs to the specified directory.

hiring_calibrator.py

Purpose: Analyzes interview scores from multiple candidates and interviewers to detect bias, calibration issues, and inconsistent rubric application. Generates calibration reports with recommendations for interviewer coaching and process improvements.

Usage:

python hiring_calibrator.py --input interview_results.json --analysis-type comprehensive --output report.json

Flags/Parameters:

Example:

python hiring_calibrator.py --input q1_interviews.json --analysis-type bias --competencies technical,leadership --trend-analysis --period quarterly --format json --output calibration/q1_bias.json

Output Formats:

• JSON: Structured calibration report containing score distributions, interviewer deviation metrics, bias indicators, trend data (if enabled), and prioritized coaching recommendations.
• Text: Human-readable report with summary statistics, flagged interviewers, bias findings, and actionable improvement recommendations formatted for management review.
• Both (default): Writes both JSON and text outputs to the specified path.