Paper Review
A systematic approach to self-reviewing academic papers before submission. Covers a 5-aspect review checklist, reverse-outlining for structural clarity, figure/table quality checks, and rebuttal preparation.
When to Use This Skill
- User wants to review or check a paper draft before submission
- User asks for feedback on paper quality or completeness
- User wants to prepare for potential reviewer criticism
- User mentions "review paper", "check my draft", "self-review"
If the user has already received reviewer comments and needs to write a rebuttal, use the paper-rebuttal skill instead.Prerequisites
Before starting review, confirm the paper-writing handoff checklist is satisfied: all sections drafted, claims anchored to evidence, limitation section present, figures finalized, and no unresolved \todo{} markers. If any item is incomplete, finish writing before reviewing.
The Perfectionist Approach
Strive for perfection: review your own paper, consider every question a reviewer might ask, and address them one by one.
The best defense against negative reviews is a thorough self-review:
- Adversarial review: Read your own paper as a critical reviewer would
- Seek advisor feedback: Ask your advisor to review — the more feedback, the better
- Address everything: For every potential weakness you find, either fix it or prepare a defense
Counterintuitive Review Protocol
Run this protocol before final polishing:
- Reject-first simulation: Force yourself to write a one-paragraph reject summary before writing any positive comments.
- Delete one unsupported strong claim: If a strong claim lacks direct evidence, remove it instead of defending it.
- Score trust, not only score gains: Papers with slightly lower gains but higher fairness and reproducibility often receive better review outcomes.
- Promote one explicit limitation: Move one meaningful limitation from hidden notes into the paper; transparency can increase confidence.
- Attack your novelty claim: Ask "Could a strong PhD derive this in one afternoon?" If yes, narrow and sharpen the novelty statement.
See references/counterintuitive-review.md
5-Aspect Self-Review Checklist
Aspect 1: Contribution Sufficiency
The paper does not provide readers with new knowledge.
Ask these questions to evaluate whether the contribution is sufficient:
- Are the failure cases common? If the failure cases are frequent and obvious, reviewers may question whether the method is ready for publication.
- Is the proposed technique well-explored? If the technique is already widely studied, what new insight or improvement do we bring?
- Is the improvement foreseeable / well-known? If the improvement was predictable from combining known ideas, the novelty may be questioned.
- Is the technique too straightforward? A straightforward application of existing techniques may lack sufficient contribution.
Red flag: If "yes" to any of these, strengthen the contribution narrative or add more technical depth.
Aspect 2: Writing Clarity
Missing technical details, not reproducible; a method module lacks motivation.
- Missing technical details? Would a reader be able to reproduce the method from the paper alone?
- Missing module motivation? Does every module in the Method section explain *why* it exists, not just *what* it does?
- Paragraph structure: Does each paragraph have a clear topic? Does the first sentence state the point?
- Flow: Is the logical flow between paragraphs and sections smooth?
- Terminology: Are terms used consistently throughout?
Red flag: If reproducibility is in doubt, add implementation details or supplementary material.
Aspect 3: Experimental Results Quality
Only slightly better than previous methods; or better than previous methods but still not good enough.
- Marginal improvement? If the improvement over SOTA is very small, is it statistically significant?
- Absolute quality insufficient? Even if better than baselines, is the output quality good enough for the application?
- Visual quality: Do qualitative results look convincing? Are improvements visible?
Red flag: If improvements are marginal, emphasize other advantages (speed, generalizability, simplicity) or add more challenging test cases.
Aspect 4: Experimental Testing Completeness
Missing ablation studies; missing important baselines; missing important evaluation metrics; data too simple.
- Missing ablation studies? Is every core contribution ablated?
- Missing important baselines? Are recent SOTA methods included?
- Missing evaluation metrics? Are all standard metrics for this task reported?
- Datasets too simple? Do the benchmarks truly test the method's capabilities?
- No failure case analysis? Honest failure analysis increases credibility.
Red flag: Missing ablations or baselines is one of the most common reasons for rejection.
Aspect 5: Method Design Issues
Experimental setting is impractical; method has technical flaws; method is not robust; new method's costs outweigh its benefits.
- Impractical experimental setting? Are assumptions realistic for the intended use case?
- Technical flaws? Does the method have theoretical or conceptual weaknesses?
- Not robust? Does the method require per-scene hyperparameter tuning?
- Benefit < Limitation? Does the new module introduce limitations that outweigh its benefits?
Red flag: If the method requires significant tuning per scenario, add robustness experiments or acknowledge and address the limitation.
Critical Reminder: Claims Must Have Support
Every claim in the paper (especially in the Abstract and Introduction) must be correct and supported by experiments. Some reviewers will reject a paper directly for unsupported claims.
Go through every claim in the Abstract and Introduction. For each claim:
- Is it factually correct?
- Is there an experiment or analysis that supports it?
- Is the supporting experiment clearly referenced?
An unsupported claim — especially in the Abstract or Introduction — can be grounds for rejection.
Reverse-Outlining Technique
Extract the writing plan from finished paragraphs and check whether the flow is smooth.
After writing a section (or the entire paper):
- Read each paragraph one at a time
- Write down the main message of each paragraph in one sentence
- Read the sequence of messages — does it flow logically?
- Identify breaks: Where does the flow feel abrupt or illogical?
- Fix: Reorganize paragraphs, add transitions, or split/merge paragraphs
Apply this to:
- Introduction (check narrative flow)
- Method (check if modules are presented in logical order)
- Experiments (check if results are presented in a meaningful sequence)
Figure and Table Quality Checklist
Figures
- Pipeline figure highlights novelty (not just explanation)
- Pipeline figure looks distinct from prior work
- Teaser figure is compelling and self-contained
- All figures have clear captions
- Resolution is high enough for print
- Color-blind friendly (avoid red-green only distinctions)
- Figures are referenced in the text
Tables
- Captions are above the table
- No vertical lines
- Using booktabs (
\toprule,\midrule,\bottomrule) - Best results highlighted (bold/color)
- Metric direction indicated (↑/↓)
- Captions describe setup/notation, not results
- All tables are referenced in the text
Conclusion and Limitation Check
- Conclusion summarizes contributions and key results
- Limitation section is present (reviewers frequently flag its absence)
- Limitations are about task/setting scope (like future work), not technical defects Rule: "If our method does not fall below SOTA metrics, it is not a technical defect"
- Limitations are honest but not self-defeating
Pre-Submission Final Checks
- All references are complete (no "?" or missing entries)
- Author information matches venue requirements
- Page count is within limits
- Supplementary material is properly referenced
- No TODO markers remain in the paper
- Acknowledgments section is appropriate
- No accidental double-blind violations (for anonymous review)
- All cited works have complete bibliographic entries (authors, title, venue, year)
- No self-citations that break anonymity (for double-blind venues)
- Key related works cited — missing a prominent baseline paper can trigger rejection
Handoff to Rebuttal
When reviews come back, use the paper-rebuttal skill for:
- Score diagnosis and review color-coding
- Champion strategy (arming your positive reviewer for discussion)
- 18 tactical rules for structure, content, and tone
- Counterintuitive rebuttal principles
Your self-review artifacts (reject-first simulation, claim-evidence audit, prebuttal drafts from the counterintuitive protocol) feed directly into the rebuttal process.
See references/review-checklist.md for an expanded version of the 5-aspect checklist with more detailed sub-questions.
For adversarial stress testing and reject-risk thresholds, see references/counterintuitive-review.md.