document-ingestion（文档摄取）

Document Ingestion Engine — SKILL.md

When to Use This Skill

Use when a user needs to process raw accounting source documents into standardized transaction records for QBO import. Triggers on:

• "Process these documents / invoices / receipts / bank statements"
• "Ingest docs for [client]"
• "I have PDFs/CSVs to categorize"
• "Batch import these transactions to QBO"
• "Extract data from 1099s / payroll reports"
• Document drop + categorization requests during month-end close

When NOT to Use

• Not for running bank reconciliation (use bank-reconciliation skill)
• Not for P&L variance analysis (use pl-quick-compare skill)
• Not for single manual journal entries (just post directly in QBO)
• Not for AR collections or aging (use ar-collections-agent skill)

What It Does

Processes 6 document types → standardized records → Excel workbook + QBO import CSV.

*Image OCR requires tesseract installed.

Processing Steps

1. File type detection — magic bytes + extension fallback
2. Document classification — bank/CC/invoice/receipt/1099/payroll
3. Content extraction — CSV parsing, OFX parsing, PDF text extraction
4. Format normalization — dates (multi-format), amounts (Decimal), vendor names (strip noise)
5. QBO COA pull — fetches live Chart of Accounts from QBO for categorization
6. Duplicate detection — same amount + vendor within ±3 days → flagged
7. Auto-categorization — vendor map → COA keywords → doc-class default
8. Confidence scoring — HIGH (exact match) / MEDIUM (fuzzy) / LOW (needs review)
9. Exception flagging — missing dates, zero amounts, unknown vendors, LOW confidence
10. QBO import CSV — ready for batch import (excludes dups + failed extractions)
11. Excel workbook — 6 tabs (see below)
12. CDC tracking — delta since last run cached in .cache/document-ingestion/{slug}.json

Excel Output Tabs

Script Location

scripts/pipelines/document-ingestion.py

Usage

# Process a directory of mixed documents
python3 scripts/pipelines/document-ingestion.py \
    --slug sb-paulson \
    --input-dir ~/Downloads/month-end-docs

# Single file
python3 scripts/pipelines/document-ingestion.py \
    --slug sb-paulson \
    --file ~/Downloads/invoice_march.pdf

# Multiple files + custom output dir
python3 scripts/pipelines/document-ingestion.py \
    --slug glowlabs \
    --file ~/Downloads/stmt.csv \
    --file ~/Downloads/payroll.csv \
    --out ~/Desktop/ingested

# Offline mode (no QBO auth needed)
python3 scripts/pipelines/document-ingestion.py \
    --slug sb-paulson \
    --input-dir ./docs \
    --no-qbo-coa

# QBO sandbox
python3 scripts/pipelines/document-ingestion.py \
    --slug sb-paulson \
    --input-dir ./docs \
    --sandbox

All CLI Flags

Dependencies

Required (pip)

pip install openpyxl

Optional (better extraction quality)

pip install pdfminer.six   # Better PDF text extraction
pip install ofxparse       # Better OFX/QFX parsing
brew install tesseract     # Image receipt OCR (JPG/PNG)

Node.js QBO Client

Node.js QBO client   # Auth token must be configured

Categorization Logic

Priority Chain

1. Vendor Map exact match → HIGH confidence
2. Vendor Map substring match → HIGH confidence
3. COA keyword index (built from COA account names + keywords) → MEDIUM confidence
4. Doc-class default → LOW confidence

Built-in Vendor Map

50+ known vendors pre-mapped:

• Stripe/Square/PayPal → Sales Revenue
• Gusto/ADP/Deel/Paychex → Payroll - Salaries & Wages
• Google/Microsoft/Slack/GitHub/Zoom → Software & Subscriptions
• Delta/United/Marriott/Uber → Travel
• FedEx/UPS/USPS → Postage & Delivery
• Chase/BofA service charges → Bank & Merchant Fees
• etc. (see VENDOR_MAP in script)

Client-Specific Overrides

Auto-loaded by --slug:

• glowlabs → Loads GlowLabs vendor map (Deel, Toptal, Brex, Huellas Labs, etc.)
• sb-paulson / willo → Loads Willo Salons vendor map
• Other clients → Reads clients/{slug}/categorization-map*.md markdown tables

Duplicate Detection Rules

• Window: ±3 days (configurable via DUP_WINDOW_DAYS constant)
• Match criteria: Same amount (exact Decimal) + same vendor key (first 3 meaningful words)
• Action: Flagged as is_duplicate=True, excluded from import file
• Always confirm before deleting — duplicates tab shows "Dup Of Row #" reference

Exception Rules (auto-flagged)

QBO Import CSV Format

Ready-to-import columns:

Date | Description | Amount | Vendor/Customer | Account | Class | Memo | Doc Number

• Amount sign: positive = expense (debit), negative = credit/income
• Memo includes source file + doc type for audit trail
• Excludes: duplicates, failed extractions

CDC Cache

Location: .cache/document-ingestion/{slug}.json

Tracks between runs:

• docs_processed, records_extracted, duplicates_caught
• exceptions_flagged, import_ready
• high_confidence, medium_confidence, low_confidence

Output File Naming

DocIngestion_{slug}_{YYYYMMDD}.xlsx
DocIngestion_{slug}_{YYYYMMDD}_QBO_Import.csv

Agent Instructions

Standard Run

1. Collect input files from user (directory path or individual files)
2. Get client slug (sb-paulson, glowlabs, etc.)
3. Run pipeline. If QBO auth not set, use --no-qbo-coa
4. Deliver summary:

- Records extracted, dups caught, exceptions - HIGH/MED/LOW confidence split - Path to Excel + import CSV

1. Walk user through Exceptions tab — those need action before import

Month-End Close Integration

• Run AFTER bank statement download, BEFORE bank reconciliation
• Use --input-dir pointing to client's document drop folder
• Import CSV goes into QBO → then run bank-reconciliation.py

Exception Handling

• PDFs with no extractable text → LOW confidence + exception flag → send to client for re-scan
• Image receipts with no tesseract → exception flag → use nano-pdf skill or manual entry
• Unknown vendors → update VENDOR_MAP in script or add to clients/{slug}/categorization-map.md

Adding New Client Vendor Maps

Edit load_client_vendor_map() in the script:

if slug_lower in ("new-client", "nc"):
    client_map.update({
        "vendor name": "QBO Account Name",
    })

Or create clients/{slug}/categorization-map.md with markdown table:

| Vendor / Memo Keyword | Primary Account | Notes |
|---|---|---|
| Amazon | Office Supplies | |
| Comcast | Utilities | |

Financial Math

All amounts use Python Decimal with ROUND_HALF_UP to 2 decimal places. No float arithmetic.