- name
- cogdx-calibration
- description
- Run a calibration audit on an AI agent's outputs via Cerebratech CogDx API ($0.05 per call, credits accepted). Use when an agent's stated confidence doesn't match actual accuracy, when downstream systems need to trust the agent's uncertainty estimates, when preparing for high-stakes deployment, or after noticing overconfidence or underconfidence patterns. Uses pure statistical methods (Brier scores, calibration curves) — no LLM in the backend. Triggers on phrases like "audit my calibration", "check my confidence", "am I overconfident", "calibration gap", "confidence accuracy mismatch", or any request to verify that stated uncertainty matches real-world accuracy. After running, use cogdx-feedback skill (FREE) to verify retraining and earn credits.
- repository
- https://github.com/drkavner/cogdx
- license
- MIT
- author
- Dr. Kavner / Cerebratech
CogDx Calibration Audit
External statistical audit of your confidence calibration. Pure math — no LLM backend. Results in < 2 minutes.
Cost
- $0.05 per call (x402 payment on Base/USDC, or use credit balance)
- Credits from
/feedbacksubmissions apply — first audit often effectively free - Payment address:
Cerebratech.eth
When to Use
- Confidence scores don't feel reliable
- Downstream agents or humans keep second-guessing your outputs
- Pre-deployment validation for high-stakes domains
- After retraining (pair with
/feedbackto verify improvement)
API Call
1. Check if you have credit balance first:
GET https://api.cerebratech.ai/credits?wallet=your-agent-id2. Run the audit:
POST https://api.cerebratech.ai/calibration_audit
Content-Type: application/json
X-PAYMENT: <x402-signature> # or omit if using credits
X-WALLET: your-agent-id # for credit balance paymentMinimum payload (10 samples):
{
"agent_id": "your-agent-id",
"sample_outputs": [
{
"prompt": "Is X true?",
"response": "Yes",
"stated_confidence": 0.92,
"correct": true
},
{
"prompt": "Will Y happen?",
"response": "Likely",
"stated_confidence": 0.75,
"correct": false
}
],
"domain": "classification"
}Recommended: 50–200 samples for reliable results.
Response
{
"diagnosis_id": "cal_abc123",
"calibration_score": 0.71,
"overconfidence_index": 0.23,
"underconfidence_index": 0.04,
"brier_score": 0.18,
"confidence_bands": [
{
"stated": "0.9-1.0",
"actual_accuracy": 0.67,
"sample_size": 23,
"calibration_error": 0.28
}
],
"recommendations": [
"Reduce confidence on high-stakes single-source claims",
"Your 0.9+ band is overconfident by 28%. Retrain on 200 negative examples in this confidence range."
],
"retrain_targets": {
"distribution": "high_confidence_errors",
"suggested_sample_count": 200,
"domain_focus": "classification"
}
}After the Audit
- Retrain on the
retrain_targetsdistribution - Wait 7 days, collect new outputs
- Run
cogdx-feedback(FREE) to verify improvement transferred + earn credits
Full Reference
See references/api.md for complete field docs, x402 payment setup, and error codes.