CodeRabbit Observability
Overview
Monitor CodeRabbit AI code review effectiveness, review latency, and team adoption. Key metrics include time-to-first-review (how fast CodeRabbit posts its review after PR creation), comment acceptance rate (comments resolved vs dismissed), review coverage (percentage of PRs reviewed), and per-repository review volume.
Prerequisites
- CodeRabbit Pro or Enterprise plan
- GitHub or GitLab organization with CodeRabbit installed
- Access to CodeRabbit dashboard at app.coderabbit.ai
Instructions
Step 1: Track Review Metrics via GitHub Webhooks
// coderabbit-metrics.ts - Process GitHub webhook events
function handlePullRequestReview(event: any) {
if (event.review.user.login === 'coderabbitai') {
const prCreatedAt = new Date(event.pull_request.created_at);
const reviewPostedAt = new Date(event.review.submitted_at);
const timeToReview = (reviewPostedAt.getTime() - prCreatedAt.getTime()) / 1000; # 1000: 1 second in ms
emitHistogram('coderabbit_time_to_review_sec', timeToReview, { repo: event.repository.name });
emitCounter('coderabbit_reviews_total', 1, { repo: event.repository.name, state: event.review.state });
}
}Step 2: Measure Comment Acceptance Rate
# Query GitHub API for CodeRabbit review comments and their resolution status
gh api repos/ORG/REPO/pulls/comments \
--jq '[.[] | select(.user.login=="coderabbitai")] | {
total: length,
resolved: [.[] | select(.resolved == true)] | length
} | {total, resolved, acceptance_rate: (.resolved / .total * 100)}'Step 3: Monitor Review Coverage
// Check what percentage of PRs get CodeRabbit reviews
async function reviewCoverage(org: string, repo: string, days: number) {
const prs = await github.pulls.list({ owner: org, repo, state: 'closed', per_page: 100 });
let reviewed = 0;
for (const pr of prs) {
const reviews = await github.pulls.listReviews({ owner: org, repo, pull_number: pr.number });
if (reviews.data.some(r => r.user?.login === 'coderabbitai')) reviewed++;
}
return { total: prs.length, reviewed, coverage: (reviewed / prs.length * 100).toFixed(1) };
}Step 4: Set Up Alerts
groups:
- name: coderabbit
rules:
- alert: CodeRabbitReviewSlow
expr: histogram_quantile(0.95, rate(coderabbit_time_to_review_sec_bucket[1h])) > 600 # 600: timeout: 10 minutes
annotations: { summary: "CodeRabbit P95 review time exceeds 10 minutes" }
- alert: CodeRabbitNotReviewing
expr: rate(coderabbit_reviews_total[6h]) == 0 and rate(github_prs_opened_total[6h]) > 0
annotations: { summary: "PRs opening but CodeRabbit not reviewing -- check installation" }
- alert: LowCommentAcceptance
expr: coderabbit_comment_acceptance_rate < 30
for: 7d
annotations: { summary: "CodeRabbit comment acceptance rate below 30% -- review config" }Step 5: Build a Dashboard
Key panels: review latency distribution, comment acceptance rate over time, PRs reviewed vs total PRs (coverage), reviews per repository (bar chart), and most common comment categories (helps tune .coderabbit.yaml instructions).
Error Handling
| Issue | Cause | Solution |
|---|---|---|
| No reviews posting | GitHub App lost permissions | Reinstall CodeRabbit App on the org |
| Review delayed >15 min | Large PR or service backlog | Expected for 1000+ line PRs; split PRs |
| Low acceptance rate | Reviews too nitpicky | Change profile from nitpicky to assertive |
| Missing repos in metrics | Repos not added to App | Add repos in GitHub App installation settings |
Examples
Basic usage: Apply coderabbit observability to a standard project setup with default configuration options.
Advanced scenario: Customize coderabbit observability for production environments with multiple constraints and team-specific requirements.
Output
- Configuration files or code changes applied to the project
- Validation report confirming correct implementation
- Summary of changes made and their rationale
Resources
- Official monitoring documentation
- Community best practices and patterns
- Related skills in this plugin pack