dedupe-rank（重复数据删除等级）

Dedupe + Rank

Turn a broad retrieved set into a smaller core set for taxonomy/outline building.

This is a deterministic “curation” step: it should be stable and repeatable.

Load Order

Always read:

• references/domain_pack_overview.md — how domain packs drive topic-specific behavior

Domain packs (loaded by topic match):

• assets/domain_packs/llm_agents.json — pinned classics, survey detection, ranking signals for LLM agent topics

Script Boundary

Use scripts/run.py only for:

• title normalization and deduplication logic
• relevance scoring from query tokens
• core set CSV generation with stable paper_id values

Do not treat run.py as the place for:

• hardcoded pinned paper IDs (use domain packs)
• hardcoded survey detection rules (use domain packs)
• domain-specific topic detection logic (use domain packs)

Input

• papers/papers_raw.jsonl

Outputs

• papers/papers_dedup.jsonl
• papers/core_set.csv

Workflow (high level)

1. Dedupe by normalized (title, year) and keep the richest metadata per duplicate cluster.
2. Rank by relevance/recency signals (and optionally pin known classics for certain topics). For LLM-agent topics, also ensure a small quota of prior surveys/reviews is present to support a paper-like Related Work section.
3. Write papers/core_set.csv with stable paper_id values and useful metadata columns (arxiv_id, pdf_url, categories).

Quality checklist

• papers/papers_dedup.jsonl exists and is valid JSONL.
• papers/core_set.csv exists and has a header row.

Script

Quick Start

• python.codex/skills/dedupe-rank/scripts/run.py --help
• python.codex/skills/dedupe-rank/scripts/run.py --workspace <workspace_dir> --core-size 300

All Options

• --core-size <n>: target size for papers/core_set.csv
• queries.md also supports core_size / core_set_size / dedupe_core_size (overrides default when present)

Examples

• Smaller core set for fast iteration (non-A150++):

- python.codex/skills/dedupe-rank/scripts/run.py --workspace <ws> --core-size 25

Notes

• This step may annotate papers/core_set.csv:reason with tags such as pinned_classic and prior_survey (deterministic, topic-aware guards for survey writing).
• Evidence-review default: if the active pipeline is evidence-review (or legacy systematic-review) and core_size is not specified, the script keeps the full deduped pool in papers/core_set.csv so screening does not silently drop candidates.
• This step is deterministic; reruns should be stable for the same inputs.

Troubleshooting

Common Issues

Issue: papers/core_set.csv is too small / empty

Symptom:

• Core set has very few rows.

Causes:

• Input papers/papers_raw.jsonl is small, or many rows are missing required fields.

Solutions:

• Broaden retrieval (or provide a richer offline export) and rerun.
• Lower --core-size only if you intentionally want a small core set.

Issue: Duplicates still appear after dedupe

Symptom:

• Near-identical titles remain.

Causes:

• Title normalization is defeated by noisy exports.

Solutions:

• Clean title fields in the export (strip prefixes/suffixes, fix encoding) and rerun.

Recovery Checklist

• papers/papers_raw.jsonl lines contain title/year/url.
• papers/core_set.csv has stable paper_id values.