post-deployment-verifier（部署后验证程序）

Post-Deployment Verifier

After deploying, verify everything works before calling it done. Checks health endpoints, validates response schemas match expectations, monitors error rates, verifies database state, and produces a confidence score.

Use when: "verify the deployment", "is the deploy healthy", "post-deploy checks", "deployment smoke test", "verify production", "deploy confidence check", or after any release.

Commands

1. verify — Full Post-Deployment Verification

Run all checks and produce a deployment confidence report.

Check 1: Health Endpoints

echo "=== Health Check ==="

# Discover health endpoints from config
ENDPOINTS=""

# From environment variables
for var in $(env | grep -iE "URL|HOST|ENDPOINT|SERVICE" | cut -d= -f2); do
  if echo "$var" | grep -qE "^https?://"; then
    ENDPOINTS="$ENDPOINTS $var/health $var/healthz $var/ready $var/status"
  fi
done

# From docker-compose
if [ -f "docker-compose.yml" ] || [ -f "docker-compose.yaml" ]; then
  PORTS=$(grep -oP 'ports:\s*\
\s*-\s*"\K[0-9]+' docker-compose*.yml 2>/dev/null | head -5)
  for port in $PORTS; do
    ENDPOINTS="$ENDPOINTS http://localhost:$port/health http://localhost:$port/healthz"
  done
fi

# From kubernetes
rg -o "readinessProbe:.*path:\s*(\S+)" -g '*.yaml' -g '*.yml' 2>/dev/null | head -5

# Check each endpoint
for endpoint in $ENDPOINTS; do
  RESPONSE=$(curl -s -o /tmp/health_body -w "%{http_code} %{time_total}s" --connect-timeout 5 --max-time 10 "$endpoint" 2>/dev/null)
  HTTP_CODE=$(echo "$RESPONSE" | awk '{print $1}')
  TIME=$(echo "$RESPONSE" | awk '{print $2}')

  if [ "$HTTP_CODE" = "200" ]; then
    echo "✅ $endpoint → $HTTP_CODE (${TIME})"
  elif [ "$HTTP_CODE" = "000" ]; then
    echo "❌ $endpoint → UNREACHABLE"
  else
    echo "⚠️  $endpoint → $HTTP_CODE (${TIME})"
    cat /tmp/health_body 2>/dev/null | head -3
  fi
done

Check 2: Response Validation

echo ""
echo "=== Response Validation ==="

# Check key API endpoints return expected structure
# Read endpoints from a .deploy-verify.json config file if it exists
if [ -f ".deploy-verify.json" ]; then
  python3 -c "
import json, urllib.request, sys

config = json.load(open('.deploy-verify.json'))
for check in config.get('endpoints', []):
    url = check['url']
    expected_status = check.get('status', 200)
    expected_fields = check.get('fields', [])
    method = check.get('method', 'GET')

    try:
        req = urllib.request.Request(url, method=method)
        for k, v in check.get('headers', {}).items():
            req.add_header(k, v)
        resp = urllib.request.urlopen(req, timeout=10)
        status = resp.status
        body = json.loads(resp.read())

        if status != expected_status:
            print(f'⚠️  {method} {url} → {status} (expected {expected_status})')
        else:
            missing = [f for f in expected_fields if f not in body]
            if missing:
                print(f'⚠️  {method} {url} → {status} OK but missing fields: {missing}')
            else:
                print(f'✅ {method} {url} → {status}, all expected fields present')
    except Exception as e:
        print(f'❌ {method} {url} → {e}')
" 2>/dev/null
else
  echo "No .deploy-verify.json config found. Create one for automated response validation."
  echo "Example:"
  echo '  {"endpoints": [{"url": "https://api.example.com/v1/status", "status": 200, "fields": ["version", "status"]}]}'
fi

Check 3: Version Verification

echo ""
echo "=== Version Check ==="

# Get expected version from package.json or git tag
EXPECTED_VERSION=""
if [ -f "package.json" ]; then
  EXPECTED_VERSION=$(python3 -c "import json; print(json.load(open('package.json')).get('version',''))" 2>/dev/null)
fi
[ -z "$EXPECTED_VERSION" ] && EXPECTED_VERSION=$(git describe --tags --abbrev=0 2>/dev/null | sed 's/^v//')
echo "Expected version: ${EXPECTED_VERSION:-unknown}"

# Check deployed version via health/status endpoint
for endpoint in $ENDPOINTS; do
  BODY=$(curl -s --connect-timeout 5 "$endpoint" 2>/dev/null)
  DEPLOYED=$(echo "$BODY" | python3 -c "import json,sys; d=json.load(sys.stdin); print(d.get('version',d.get('app_version',d.get('build','?'))))" 2>/dev/null)
  if [ -n "$DEPLOYED" ] && [ "$DEPLOYED" != "?" ]; then
    if [ "$DEPLOYED" = "$EXPECTED_VERSION" ]; then
      echo "✅ Deployed version matches: $DEPLOYED"
    else
      echo "⚠️  Version mismatch: deployed=$DEPLOYED, expected=$EXPECTED_VERSION"
    fi
  fi
done

Check 4: Error Rate Monitoring

echo ""
echo "=== Error Monitoring ==="

# Check recent application logs for errors
if command -v docker &>/dev/null; then
  echo "--- Docker container logs (last 5 min) ---"
  for container in $(docker ps --format "{{.Names}}" 2>/dev/null); do
    ERRORS=$(docker logs --since 5m "$container" 2>&1 | grep -ciE "error|exception|fatal|panic" 2>/dev/null)
    if [ "$ERRORS" -gt 0 ]; then
      echo "⚠️  $container: $ERRORS errors in last 5 minutes"
      docker logs --since 5m "$container" 2>&1 | grep -iE "error|exception|fatal|panic" | tail -3
    else
      echo "✅ $container: no errors"
    fi
  done
fi

# Check systemd service logs
if command -v journalctl &>/dev/null; then
  echo "--- Systemd service logs (last 5 min) ---"
  for svc in $(systemctl list-units --type=service --state=running --no-legend 2>/dev/null | awk '{print $1}' | grep -v systemd); do
    ERRORS=$(journalctl -u "$svc" --since "5 min ago" --no-pager -q 2>/dev/null | grep -ciE "error|exception|fatal" 2>/dev/null)
    [ "$ERRORS" -gt 0 ] && echo "⚠️  $svc: $ERRORS errors"
  done
fi

Check 5: Database Migration Verification

echo ""
echo "=== Database State ==="

# Check if migrations are up to date
if [ -f "package.json" ]; then
  # Prisma
  rg -q "prisma" package.json 2>/dev/null && echo "Prisma detected — run: npx prisma migrate status"
  # TypeORM
  rg -q "typeorm" package.json 2>/dev/null && echo "TypeORM detected — run: npx typeorm migration:show"
  # Sequelize
  rg -q "sequelize" package.json 2>/dev/null && echo "Sequelize detected — run: npx sequelize db:migrate:status"
fi

# Python/Django
[ -f "manage.py" ] && echo "Django detected — run: python manage.py showmigrations"

# Rails
[ -f "Gemfile" ] && grep -q "rails" Gemfile 2>/dev/null && echo "Rails detected — run: rails db:migrate:status"

# Check for pending migration files
PENDING=$(find . -path "*/migrations/*" -name "*.sql" -newer .last-deploy-timestamp 2>/dev/null | wc -l)
[ "$PENDING" -gt 0 ] && echo "⚠️  $PENDING migration files newer than last deploy"

Check 6: Resource Utilization

echo ""
echo "=== Resource Check ==="

# Docker resource usage
if command -v docker &>/dev/null; then
  docker stats --no-stream --format "table {{.Name}}\	{{.CPUPerc}}\	{{.MemUsage}}\	{{.NetIO}}" 2>/dev/null | head -10
fi

# System resources
echo "--- System ---"
free -h | head -2
df -h / | tail -1
uptime

2. quick — Quick Health Check

Run only health endpoint checks (Check 1) for fast verification. Use immediately after deploy.

3. config — Generate Verification Config

Create a .deploy-verify.json file with endpoints to check:

{
  "endpoints": [
    {
      "url": "https://api.example.com/health",
      "status": 200,
      "fields": ["status", "version"]
    },
    {
      "url": "https://api.example.com/v1/users",
      "method": "GET",
      "status": 200,
      "headers": {"Authorization": "Bearer ${API_TOKEN}"},
      "fields": ["data", "pagination"]
    }
  ],
  "thresholds": {
    "max_response_time_ms": 500,
    "max_error_rate_percent": 1,
    "min_uptime_percent": 99.9
  }
}

4. report — Deployment Confidence Report

# Deployment Verification Report
Date: [date] | Version: [version] | Environment: [env]

## Confidence Score: 92/100 🟢

| Check | Status | Details |
|-------|--------|---------|
| Health endpoints | ✅ Pass | 3/3 healthy |
| Response schema | ✅ Pass | All fields present |
| Version match | ✅ Pass | v1.5.0 deployed |
| Error rate | ⚠️ Warning | 2 errors in 5 min |
| Database | ✅ Pass | All migrations applied |
| Resources | ✅ Pass | CPU 12%, Mem 45% |

## Action Required
- Monitor error rate for next 30 minutes (currently elevated)

## Recommendation: KEEP (no rollback needed)

Confidence scoring:

• 95-100: All clear, no action needed
• 80-94: Minor warnings, monitor closely
• 60-79: Issues found, consider rollback if degrading
• <60: Significant problems, recommend rollback

Output Formats

• text (default): Human-readable with status icons
• json: {confidence: 92, checks: [{name, status, details}], recommendation: "KEEP"}
• markdown: Stakeholder-ready report
• slack: Formatted for Slack webhook delivery

Notes

• Run immediately after deployment for maximum value
• Config file (.deploy-verify.json) enables consistent, repeatable verification
• Does not require access to monitoring systems — uses direct endpoint checks
• For production: run from a separate monitoring instance, not the deployed service itself
• Complement with Datadog/Grafana dashboards for continuous monitoring
• Supports environment variable substitution in config (${VAR} syntax)