agent-causal（动因因果）

Agent Causal Decision Tool

What this skill does

Agent Causal Decision Tool turns experiment or rollout data into a clear, defensible decision: ship, keep running, or roll back. It takes simple A/B or rollout summaries and returns structured JSON with:

• A recommended decision and next action.
• Key statistics (rates, lift, probabilities, DiD estimate).
• Diagnostics, warnings, and an audit trail for humans to review later.

You bring the data (from logs, BI, CSV); the tool handles the statistics, decision logic, and audit record.

Why it exists

Most teams still judge experiments in spreadsheets or dashboards, arguing over noisy lifts and sample size. Agents can make this worse if they react to any small change.

This skill wraps standard methods behind a consistent, agent‑friendly interface:

• Frequentist A/B testing for classic control vs variant.
• Bayesian A/B testing when you want answers like "there is a 93% chance B is better than A."
• Difference‑in‑Differences (DiD) for staged or regional rollouts where you cannot fully randomize.
• Planning and power checks to see if a test is realistic before you start.
• Decision audit and history so humans can see what the agent did, why, and how strong the evidence was..
• Sequential / early stopping — opt‑in flag to stop A/B tests early when evidence is clearly strong, with conservative thresholds and full audit trail..
• Cohort / segment breakdown — when an aggregate A/B result is inconclusive, break it down by user segment to find hidden signals. Uses Benjamini‑Hochberg correction for 4+ segments..

It is not a full experimentation platform; it's a small, reliable decision block that agents can call inside workflows.

When to use it

Use this skill whenever you or your agents have experiment or rollout results and need a decision you can defend:

• You ran an A/B test and want to know whether to ship, keep running, or reject the variant.
• You ran an A/B test and it was inconclusive — you want to know if a specific user segment is driving (or diluting) the effect.
• You did a staged / regional rollout and want a DiD estimate of impact vs a similar control group.
• You prefer a Bayesian summary ("95% chance B is better; expected lift 3–5%") to drive thresholds in automated workflows.
• You need an audit trail with period, traffic, assumptions, thresholds, and warnings for product/data/risk review.
• You want to plan an experiment (sample size, MDE, duration) or compare current results to previous experiments.

Quickstart

# Install
pip install git+https://github.com/ZhuMorris/agent-causal-decision-tool.git -q

# Run your first A/B decision
cd ~/clawd/agent-causal-decision-tool
PYTHONPATH=. python3 -m src.cli ab --control 100/5000 --variant 130/5000

Setup

Install the tool on the host where the agent will execute it.

Option 1: Install from Git (recommended)

pip install git+https://github.com/ZhuMorris/agent-causal-decision-tool.git -q

Option 2: Clone the repo manually

# Clone the repository (if not already present)
git clone https://github.com/ZhuMorris/agent-causal-decision-tool.git ~/clawd/agent-causal-decision-tool 2>/dev/null || true

# Install dependencies
pip install click scipy numpy pydantic -q

# Navigate to the tool directory
cd ~/clawd/agent-causal-decision-tool

Core commands

1. Experiment planning (plan)

Estimate required sample size, duration, and feasibility before running an experiment.

cd ~/clawd/agent-causal-decision-tool
PYTHONPATH=. python3 -m src.cli plan --baseline 0.02 --mde 5 --traffic 5000

Key parameters:

• --baseline (required): baseline conversion rate (e.g. 0.02 for 2%).
• --mde (required): minimum detectable effect as % lift (e.g. 5 for +5%).
• --traffic (required): daily traffic per arm.
• --confidence (default 0.95), --power (default 0.8).
• --format: json (default) or text.

Planning output (JSON):

{
  "mode": "planning",
  "recommendation": {
    "decision": "feasible | slow | not_recommended",
    "confidence": "high | medium | low",
    "summary": "..."
  },
  "planning": {
    "required_sample_per_arm": 182934,
    "total_required": 365868,
    "estimated_days": 36.6,
    "feasibility": "slow",
    "allocation_used": { "control": 0.5, "variant": 0.5 }
  },
  "warnings": [...]
}

Feasibility thresholds:

• feasible: ≤ 14 days
• slow: 15–60 days
• not_recommended: > 60 days

2. A/B test (frequentist, ab)

Classic frequentist A/B decision with rates, lift, and p‑value.

cd ~/clawd/agent-causal-decision-tool
PYTHONPATH=. python3 -m src.cli ab --control 100/5000 --variant 130/5000

Key parameters:

• --control: conversions/total for control (e.g. 100/5000).
• --variant: conversions/total for variant (e.g. 130/5000).
• --name: optional variant name (default: variant_1).
• --format: json (default) or text.

Typical JSON output (simplified):

{
  "version": "1.0",
  "mode": "ab_test",
  "recommendation": {
    "decision": "ship",
    "confidence": "medium",
    "summary": "Variant performs 30.00% better (p=0.0454). Ship it."
  },
  "statistics": {
    "control_rate": 0.02,
    "variant_rate": 0.026,
    "relative_lift_pct": 30.0,
    "p_value": 0.045361
  },
  "traffic_stats": {
    "control_size": 5000,
    "variant_size": 5000,
    "total_size": 10000
  },
  "warnings": [],
  "next_steps": ["Deploy variant", "Monitor over time for regression"],
  "next_analysis_suggestion": {
    "command": "cohort-breakdown",
    "reason": "Aggregate result is inconclusive. A segment-level breakdown may reveal hidden signal.",
    "trigger": "decision=escalate"
  },
  "audit": {
    "decision_path": [
      { "step": "Input validation", "passed": true },
      { "step": "Traffic check", "passed": true },
      { "step": "Conversion rate calculation", "passed": true },
      { "step": "Statistical significance test", "passed": true },
      { "step": "Effect size check", "passed": true },
      { "step": "Decision", "passed": true }
    ]
  }
}

3. A/B test (Bayesian, bayes)

Bayesian A/B with probability of winning and lift distribution.

cd ~/clawd/agent-causal-decision-tool
PYTHONPATH=. python3 -m src.cli bayes --control 100/5000 --variant 130/5000

Uses a Beta‑Binomial model with a Jeffreys prior:

• Prior: Beta(0.5, 0.5).
• Posterior: Beta(α + successes, β + failures).
• Monte Carlo sampling (default 20k samples) to estimate P(variant wins).
• Example thresholds: P(variant wins) ≥ 0.95 → ship, ≤ 0.05 → reject.

Key parameters:

• --control, --variant: same format as ab.
• --name: variant name.
• --format: json or text.
• --samples: number of Monte Carlo samples (default 20000).

Typical JSON output (simplified):

{
  "mode": "bayesian_ab",
  "recommendation": {
    "decision": "ship",
    "confidence": "medium",
    "summary": "Variant wins with P(better)=0.976. Median lift=30.10%."
  },
  "statistics": {
    "p_variant_wins": 0.9758,
    "lift_median_pct": 30.10,
    "lift_95ci_pct": [0.20, 69.15],
    "posterior_control": { "alpha": 100.5, "beta": 4900.5, "mean": 0.0201 },
    "posterior_variant": { "alpha": 130.5, "beta": 4870.5, "mean": 0.0261 }
  }
}

When to use which:

• Bayesian: small data, need probabilities and lift intervals, may stop early.
• Frequentist: large data, traditional p‑values, compatibility with existing practice.

4. Difference‑in‑Differences (DiD, did)

For staged or regional rollouts where you cannot fully randomize.

cd ~/clawd/agent-causal-decision-tool
PYTHONPATH=. python3 -m src.cli did \
  --pre-control 1000 --post-control 1100 \
  --pre-treated 900 --post-treated 1150

Key parameters:

• --pre-control, --post-control: control metric before/after.
• --pre-treated, --post-treated: treated metric before/after.

Typical JSON output (simplified):

{
  "mode": "did",
  "recommendation": {
    "decision": "escalate",
    "confidence": "low",
    "summary": "Effect looks positive but caution level is high — escalate for human review."
  },
  "effect": {
    "treatment_change": 0.25,
    "control_change": 0.10,
    "did_estimate": 0.15,
    "relative_did_pct": 16.67
  },
  "statistics": {
    "pre_control": 1000,
    "post_control": 1100,
    "pre_treated": 900,
    "post_treated": 1150
  },
  "diagnostics": {
    "parallel_trends_evidence": "weak",
    "fragility_flags": ["single_pre_period", "small_sample"],
    "recommended_caution_level": "high"
  },
  "warnings": [
    { "code": "did_result_should_be_reviewed_by_human", "severity": "warning" },
    { "code": "AGGREGATE_DATA", "severity": "info" }
  ],
  "next_steps": ["Escalate to human for review", "Do not treat as randomized experiment"]
}

5. Cohort / segment breakdown (cohort-breakdown)

When an aggregate A/B or DiD result is inconclusive, break it down by user segment to find hidden signals. The output flags when a segment-level result contradicts the aggregate decision.

cd ~/clawd/agent-causal-decision-tool
PYTHONPATH=. python3 -m src.cli cohort-breakdown --file segments.json

JSON input:

{
  "experiment_id": "checkout-v3",
  "metric": "conversion_rate",
  "prior_result_id": "dec_20260501_001",
  "prior_decision": "wait",
  "segments": [
    {
      "segment_name": "new_users",
      "segment_definition_note": "Users registered within last 30 days",
      "control_conversions": 21,
      "control_total": 1000,
      "variant_conversions": 67,
      "variant_total": 1000
    },
    {
      "segment_name": "returning_users",
      "segment_definition_note": "Users registered more than 30 days ago",
      "control_conversions": 220,
      "control_total": 4000,
      "variant_conversions": 228,
      "variant_total": 4000
    }
  ]
}

CSV input (alternative):

segment_name,segment_definition_note,arm,conversions,total
new_users,Users registered within last 30 days,control,21,1000
new_users,Users registered within last 30 days,variant,67,1000
returning_users,Users registered more than 30 days ago,control,220,4000
returning_users,Users registered more than 30 days ago,variant,228,4000

Typical JSON output:

{
  "method": "experiment_cohort_breakdown",
  "prior_result_id": "dec_20260501_001",
  "prior_decision": "wait",
  "cohort_decision_override": true,
  "cohort_override_reason": "Strong positive signal in 'new_users' contradicts aggregate 'wait'",
  "interaction_flag": false,
  "segments": [
    {
      "segment_name": "new_users",
      "control_rate": 0.021,
      "variant_rate": 0.067,
      "relative_lift_pct": 219.0,
      "p_value_raw": 0.000001,
      "p_value_adjusted": 0.000001,
      "decision": "strongly_positive",
      "priority_rank": 1
    },
    {
      "segment_name": "returning_users",
      "control_rate": 0.055,
      "variant_rate": 0.057,
      "relative_lift_pct": 3.6,
      "p_value_raw": 0.697,
      "p_value_adjusted": 0.697,
      "decision": "neutral",
      "priority_rank": 2
    }
  ],
  "priority_ranking": [
    { "rank": 1, "segment": "new_users", "rationale": "Strong positive effect, statistically significant after BH correction" },
    { "rank": 2, "segment": "returning_users", "rationale": "No meaningful effect detected. Deprioritize." }
  ],
  "summary": "new_users drives the effect. 1 segment(s) positive.",
  "recommended_next_action": "targeted_rollout",
  "warnings": [],
  "audit": {
    "test_type": "two_proportion_z_test",
    "multiple_comparison_method": "benjamini_hochberg",
    "total_segments_compared": 2
  }
}

Statistical method:

• Per-segment: two-proportion z-test (same as ab)
• 2–3 segments: no multiple-comparison correction
• 4+ segments: Benjamini-Hochberg (BH) FDR control by default
• 5+ segments: Bonferroni available as optional override (tool warns when used)

cohort_decision_override: Fires when a strongly_positive segment contradicts an aggregate wait/escalate decision, or when a strongly_negative segment contradicts ship. Agents should treat this as a signal to recommend targeted rollout rather than accepting the aggregate result.

interaction_flag: Fires when one segment is strongly_positive and another is strongly_negative — a possible interaction effect worth flagging.

6. Decision audit (audit)

Reconstruct and explain a previous decision.

# Save result to a file
PYTHONPATH=. python3 -m src.cli ab --control 100/5000 --variant 130/5000 > /tmp/result.json

# Human-readable audit
PYTHONPATH=. python3 -m src.cli audit /tmp/result.json --format text

# Audit with maturity assessment
PYTHONPATH=. python3 -m src.cli audit /tmp/result.json --maturity

Maturity assessment:

• Scores an experiment 0–100 across multiple checks (warnings, coverage, documentation, traffic).
• Labels: mature (≥ 90), adequate (≥ 70), immature (≥ 50), inadequate (< 50).

6. Experiment history and comparison (history, compare, save)

Persist experiment outputs to a local SQLite database and compare them.

# Run and save
PYTHONPATH=. python3 -m src.cli ab --control 100/5000 --variant 130/5000 --save
PYTHONPATH=. python3 -m src.cli did --pre-control 1000 --post-control 1100 --pre-treated 900 --post-treated 1150 --save
PYTHONPATH=. python3 -m src.cli plan --baseline 0.02 --mde 5 --traffic 5000 --save

# List recent experiments
PYTHONPATH=. python3 -m src.cli history
PYTHONPATH=. python3 -m src.cli history --mode ab_test --limit 10

# Compare experiments by ID
PYTHONPATH=. python3 -m src.cli compare 1 2 3

# Save an existing JSON result into history
PYTHONPATH=. python3 -m src.cli save /tmp/result.json --name "checkout-v3-test"

SQLite DB is stored at ~/.agent-causal/history.db. All raw JSON is preserved for later audit and comparison.

Decision reference

How to interpret the decision field in recommendations:

Warnings and agent actions

When a warning appears, here is the suggested agent response:

Python API (optional)

You can also call the underlying library directly from Python:

from src.ab_test import calculate_ab
from src.bayes import calculate_bayes_ab
from src.did import calculate_did
from src.cohort import cohort_breakdown

# Frequentist A/B
result = calculate_ab({
  "control_conversions": 100,
  "control_total": 5000,
  "variant_conversions": 130,
  "variant_total": 5000
})
if result.recommendation.decision == "ship":
  pass  # Deploy

# Bayesian A/B
result = calculate_bayes_ab({
  "control_conversions": 100,
  "control_total": 5000,
  "variant_conversions": 130,
  "variant_total": 5000
})
if result["recommendation"]["decision"] == "ship":
  pass  # Deploy

# DiD
result = calculate_did({
  "pre_control": 1000, "post_control": 1100,
  "pre_treated": 900, "post_treated": 1150
})

# Cohort breakdown
result = cohort_breakdown({
  "experiment_id": "checkout-v3",
  "metric": "conversion_rate",
  "prior_result_id": "dec_20260501_001",
  "prior_decision": "wait",
  "segments": [
    {
      "segment_name": "new_users",
      "segment_definition_note": "Users registered within last 30 days",
      "control_conversions": 21,
      "control_total": 1000,
      "variant_conversions": 67,
      "variant_total": 1000
    }
  ]
})
if result["cohort_decision_override"]:
  pass  # Targeted rollout recommended

Location and dependencies

• GitHub: https://github.com/ZhuMorris/agent-causal-decision-tool
• Local path (if cloned): ~/clawd/agent-causal-decision-tool/

Dependencies:

• Python 3.9+
• click ≥ 8.1.0
• scipy ≥ 1.11.0
• numpy ≥ 1.24.0
• pydantic ≥ 2.0.0