afrexai-qa-engine（afrexai 质量保证引擎）

QA & Test Engineering Command Center

Complete quality assurance system — from test strategy to automation frameworks, coverage analysis, and release readiness. Works for any stack, any team size.

When to Use

• Planning test strategy for a new feature or project
• Writing unit, integration, or E2E tests
• Reviewing test quality and coverage gaps
• Setting up test automation and CI/CD quality gates
• Performance testing and load analysis
• Security testing checklist
• Bug triage and defect management
• Release readiness assessment

Phase 1: Test Strategy

Strategy Brief

Before writing any tests, define the strategy:

# test-strategy.yaml
project: "[name]"
scope: "[feature/module/full product]"
risk_level: high | medium | low
stack:
  language: "[TypeScript/Python/Java/Go]"
  framework: "[React/Express/Django/Spring]"
  test_runner: "[Jest/Vitest/pytest/JUnit/Go test]"
  e2e_tool: "[Playwright/Cypress/Selenium]"

# What are we testing?
test_scope:
  - area: "[e.g., Auth module]"
    risk: high
    test_types: [unit, integration, e2e]
    priority: 1
  - area: "[e.g., Settings page]"
    risk: low
    test_types: [unit]
    priority: 3

# What's NOT in scope (and why)
exclusions:
  - "[e.g., Third-party widget — covered by vendor]"

# Quality targets
targets:
  line_coverage: 80
  branch_coverage: 70
  critical_path_coverage: 100
  max_flaky_rate: 2%
  max_test_duration_unit: 10ms
  max_test_duration_integration: 500ms
  max_test_duration_e2e: 30s

Risk-Based Test Allocation

Not everything needs the same testing depth. Use the risk matrix:

Test Pyramid

Follow the pyramid — not the ice cream cone:

         /  E2E  \          ← Few (5-10%) — slow, expensive, brittle
        / Integr. \         ← Some (15-25%) — API contracts, DB queries
       /   Unit    \        ← Many (65-80%) — fast, isolated, cheap

Anti-pattern: Ice cream cone (mostly E2E, few unit tests) = slow CI, flaky builds, expensive maintenance.

Decision rule: Can this be tested at a lower level? → Test it there.

Phase 2: Unit Testing

Anatomy of a Good Unit Test

Every unit test follows AAA (Arrange-Act-Assert):

1. ARRANGE — Set up test data, mocks, state
2. ACT     — Call the function/method under test
3. ASSERT  — Verify the output matches expectations

Unit Test Checklist (per function)

For each function/method, verify:

• [ ] Happy path — expected input → expected output
• [ ] Edge cases — empty input, null/undefined, zero, max values
• [ ] Boundary values — off-by-one, min-1, max+1
• [ ] Error handling — invalid input → correct error thrown
• [ ] Return types — correct type, shape, structure
• [ ] Side effects — does it modify state it shouldn't?
• [ ] Idempotency — calling twice gives same result?

What to Mock (and What NOT to Mock)

Mock these:

• External APIs (HTTP calls, third-party services)
• Database queries (in unit tests only)
• File system operations
• Date/time (use fake timers)
• Random number generators
• Environment variables

DO NOT mock these:

• The function under test itself
• Pure utility functions (test them directly)
• Data transformations
• Simple value objects

Mock rule of thumb: If removing the mock would make the test hit the network, file system, or database → mock it. Otherwise → don't.

Test Naming Convention

Use the pattern: [unit] [scenario] [expected result]

Examples:

• calculateTotal returns 0 for empty cart
• validateEmail throws for missing @ symbol
• parseDate handles ISO 8601 with timezone offset

Coverage Analysis

Metrics that matter:

Coverage traps to avoid:

• 100% line coverage ≠ good tests (assertions matter more than lines hit)
• Coverage on generated code inflates numbers
• Trivial getters/setters pad coverage without value
• Coverage should INCREASE over time, never decrease

Phase 3: Integration Testing

What Integration Tests Cover

Integration tests verify that components work TOGETHER:

• API endpoint → middleware → handler → database → response
• Service A calls Service B and handles the response
• Message queue producer → consumer → side effect
• Auth flow: login → token → authenticated request

Integration Test Patterns

Pattern 1: API Contract Testing

1. Start test server (or use supertest/httptest)
2. Send HTTP request with specific payload
3. Assert: status code, response body shape, headers
4. Assert: database state changed correctly
5. Assert: side effects triggered (emails, events)

Pattern 2: Database Integration

1. Start test database (SQLite in-memory or test container)
2. Run migrations
3. Seed test data
4. Execute query/operation
5. Assert: data matches expectations
6. Teardown (truncate or rollback transaction)

Pattern 3: External Service

1. Record real API response (VCR/nock/wiremock)
2. Replay recorded response in tests
3. Assert: your code handles the response correctly
4. Also test: timeout, 500 error, malformed response

Integration Test Checklist

• [ ] Happy path — full flow works end-to-end
• [ ] Auth — unauthenticated returns 401, wrong role returns 403
• [ ] Validation — bad payload returns 400 with error details
• [ ] Not found — missing resource returns 404
• [ ] Conflict — duplicate create returns 409
• [ ] Rate limiting — excessive requests return 429
• [ ] Database constraints — unique violations, foreign keys
• [ ] Concurrency — two simultaneous writes don't corrupt data
• [ ] Timeout handling — external service timeout → graceful fallback

Phase 4: End-to-End (E2E) Testing

E2E Strategy

E2E tests verify complete user journeys. They're expensive — be strategic:

Test these E2E:

• User registration → email verification → first login
• Purchase flow → payment → confirmation
• Critical business workflows (the ones that make money)
• Cross-browser/device smoke tests

DON'T test these E2E:

• Individual form validations (unit test)
• API error handling (integration test)
• Edge cases (lower-level tests)
• Visual styling (visual regression tools)

E2E Test Template

test_name: "[User journey name]"
preconditions:
  - "[User is logged in]"
  - "[Product exists in catalog]"
steps:
  - action: "Navigate to /products"
    verify: "Product list is visible"
  - action: "Click 'Add to Cart' on Product A"
    verify: "Cart badge shows 1"
  - action: "Click 'Checkout'"
    verify: "Checkout form displayed"
  - action: "Fill payment details and submit"
    verify: "Order confirmation page with order ID"
postconditions:
  - "Order exists in database with status 'paid'"
  - "Confirmation email sent"
max_duration: 30s

Flaky Test Management

Flaky tests are the #1 CI killer. Handle them:

Flaky Test Triage:

1. Identify — Track test pass rates over 10+ runs
2. Classify — Why is it flaky?

- Timing/race condition → Add explicit waits, not sleep() - Test data dependency → Isolate test data per run - External service → Mock it or use test container - Browser rendering → Use visibility checks, not delays

1. Quarantine — Move to @flaky suite, run separately
2. Fix or delete — Flaky test unfixed for 2 weeks → delete it

Flaky rate target: < 2% of total test runs

Phase 5: Performance Testing

Performance Test Types

Performance Test Plan

test_name: "[API/Page] Load Test"
target: "[URL or endpoint]"
baseline:
  p50_response: "[current p50 ms]"
  p95_response: "[current p95 ms]"
  p99_response: "[current p99 ms]"
  error_rate: "[current %]"

scenarios:
  - name: "Normal load"
    vus: 50          # virtual users
    duration: 5m
    ramp_up: 30s
    thresholds:
      p95_response: "< 500ms"
      error_rate: "< 1%"

  - name: "Peak load"
    vus: 200
    duration: 10m
    ramp_up: 1m
    thresholds:
      p95_response: "< 2000ms"
      error_rate: "< 5%"

  - name: "Stress test"
    vus: 500
    duration: 5m
    ramp_up: 2m
    # Find the breaking point — no thresholds, observe

Performance Metrics Dashboard

Track these per endpoint:

Common Performance Fixes

Phase 6: Security Testing

Security Test Checklist

Run through these for every feature/release:

Authentication & Authorization:

• [ ] Passwords hashed with bcrypt/argon2 (not MD5/SHA1)
• [ ] Session tokens are random, sufficient length (128+ bits)
• [ ] JWT tokens have short expiry (15 min access, 7 day refresh)
• [ ] Failed login rate limiting (5 attempts → lockout)
• [ ] Password reset tokens expire (1 hour max)
• [ ] Role-based access enforced server-side (not just UI)
• [ ] Can't access other users' data by changing IDs in URL

Input Validation:

• [ ] SQL injection — parameterized queries everywhere
• [ ] XSS — output encoding, CSP headers
• [ ] CSRF — tokens on state-changing requests
• [ ] Path traversal — validate file paths, no ../
• [ ] Command injection — never pass user input to shell
• [ ] File upload — validate type, size, scan for malware
• [ ] JSON/XML parsing — depth limits, entity expansion disabled

Data Protection:

• [ ] HTTPS everywhere (HSTS header)
• [ ] Sensitive data encrypted at rest
• [ ] PII not logged (mask in log output)
• [ ] API keys not in client-side code
• [ ] CORS configured correctly (not *)
• [ ] Security headers set (X-Frame-Options, X-Content-Type-Options)

Infrastructure:

• [ ] Dependencies scanned for CVEs (npm audit / pip audit)
• [ ] Docker images scanned (Trivy/Snyk)
• [ ] Secrets not in code/env files (use vault)
• [ ] Error messages don't leak internals
• [ ] Admin endpoints behind VPN/IP allowlist

OWASP Top 10 Quick Reference

Phase 7: Bug Triage & Defect Management

Bug Report Template

bug_id: "[auto or manual]"
title: "[Short description of the bug]"
severity: P0-critical | P1-high | P2-medium | P3-low
reporter: "[name]"
date: "[YYYY-MM-DD]"

environment:
  os: "[OS + version]"
  browser: "[Browser + version]"
  app_version: "[version/commit]"
  
steps_to_reproduce:
  1. "[Step 1]"
  2. "[Step 2]"
  3. "[Step 3]"

expected_result: "[What should happen]"
actual_result: "[What actually happens]"
frequency: "always | intermittent | once"
screenshots: "[links]"
logs: "[relevant log output]"

Severity Classification

Bug Triage Process (Weekly)

1. Review all new bugs (unassigned)
2. For each bug:
   a. Reproduce — can you trigger it?
   b. Classify severity (P0-P3)
   c. Estimate fix effort (S/M/L)
   d. Assign to owner + sprint
   e. Link to related bugs/stories
3. Review P0/P1 bugs from last week — are they fixed?
4. Close bugs that can't be reproduced (after 2 attempts)
5. Update metrics dashboard

Bug Metrics Dashboard

Track weekly:

Phase 8: Release Readiness

Release Checklist

Before shipping to production:

Code Quality:

• [ ] All unit tests passing
• [ ] All integration tests passing
• [ ] E2E smoke suite passing
• [ ] No new lint warnings/errors
• [ ] Code reviewed and approved
• [ ] No known P0/P1 bugs open for this release

Coverage & Quality Gates:

• [ ] Line coverage ≥ target (80%)
• [ ] Branch coverage ≥ target (70%)
• [ ] No coverage decrease from last release
• [ ] Mutation testing score ≥ 60% (if applicable)

Performance:

• [ ] Load test passed (within thresholds)
• [ ] No performance regressions vs baseline
• [ ] Bundle size within budget

Security:

• [ ] Dependency audit clean (no critical/high CVEs)
• [ ] Security checklist completed
• [ ] Secrets rotated if needed

Operational Readiness:

• [ ] Monitoring/alerts configured for new features
• [ ] Rollback plan documented
• [ ] Feature flags in place for risky changes
• [ ] Database migration tested and reversible
• [ ] Runbook updated

Release Readiness Score

Score 0-100 across 5 dimensions:

Ship threshold: ≥ 80 overall, no dimension below 60

Phase 9: CI/CD Quality Gates

Pipeline Quality Gates

Configure these gates in your CI pipeline:

# Quality gate configuration
gates:
  - name: "Lint"
    stage: pre-commit
    command: "npm run lint"
    blocking: true
    
  - name: "Unit Tests"
    stage: commit
    command: "npm test -- --coverage"
    blocking: true
    thresholds:
      pass_rate: 100%
      coverage_line: 80%
      coverage_branch: 70%
      
  - name: "Integration Tests"
    stage: merge
    command: "npm run test:integration"
    blocking: true
    thresholds:
      pass_rate: 100%
      
  - name: "Security Scan"
    stage: merge
    command: "npm audit --audit-level=high"
    blocking: true
    
  - name: "E2E Smoke"
    stage: staging
    command: "npm run test:e2e:smoke"
    blocking: true
    thresholds:
      pass_rate: 100%
      
  - name: "Performance"
    stage: staging
    command: "npm run test:perf"
    blocking: false  # Alert only
    thresholds:
      p95_regression: 20%

Test Automation Maturity Model

Rate your team 1-5:

Phase 10: Test Maintenance

Weekly Test Health Review

review_date: "[YYYY-MM-DD]"

metrics:
  total_tests: 0
  pass_rate_7d: "0%"
  flaky_tests: 0
  flaky_rate: "0%"
  avg_suite_duration: "0s"
  coverage_line: "0%"
  coverage_branch: "0%"
  
actions:
  quarantined: []     # Tests moved to flaky suite
  deleted: []         # Tests removed (obsolete/unfixable)
  fixed: []           # Flaky tests fixed this week
  added: []           # New tests added
  
trends:
  coverage_delta: "+0%"     # vs last week
  flaky_delta: "+0"         # vs last week
  duration_delta: "+0s"     # vs last week
  
notes: ""

Test Maintenance Rules

1. No commented-out tests — delete or fix, never comment
2. No skipped tests > 2 weeks — fix or remove
3. No test duplication — each behavior tested once at the right level
4. Test names must be readable — someone new should understand what broke
5. Shared test utilities — common setup in fixtures/factories, not copy-pasted
6. Test data isolation — each test creates its own data, cleans up after
7. No magic numbers — use named constants in assertions
8. Assertion messages — custom messages on complex assertions

Common Test Anti-Patterns

Quick Reference: Natural Language Commands

Tell the agent:

• "Create test strategy for [feature]" → Generates strategy brief
• "Write unit tests for [function/file]" → AAA-structured tests with edge cases
• "Review test coverage for [module]" → Gap analysis + recommendations
• "Write integration tests for [API endpoint]" → Full HTTP test suite
• "Plan E2E tests for [user journey]" → E2E test template
• "Run security checklist for [feature]" → OWASP-based security review
• "Triage these bugs: [list]" → Severity classification + assignment
• "Release readiness check" → Full readiness score + blockers
• "Performance test plan for [endpoint]" → Load/stress test configuration
• "Fix flaky test [name]" → Root cause analysis + fix strategy