- name
- data-harvester-v2
- description
- Batch web scraping for competitor analysis, price monitoring and market research
- command-dispatch
- tool
- command-tool
- data_harvester
- command-arg-mode
- raw
- metadata
- {"openclaw": {"emoji": "🧑🌾", "requires": {"bins": ["python3"]}}}
🧑🌾 Data Harvester - Batch Scraping & Competitor Analysis
Professional-grade batch web scraping tool for competitor monitoring, price tracking, and market research.
Features
- Batch URL Scraping: Process multiple URLs from file
- Competitor Comparison: Compare products across platforms
- Stock Data: Real-time stock quotes and analysis
- Sector Trends: Hot sectors and market trends
- News Aggregation: Collect news by keywords
Commands
| Command | Description | Example |
|---|---|---|
stock <code> | Query stock data | /data-harvester stock 600519 |
block | Hot market sectors | /data-harvester block |
fund | Money flow tracking | /data-harvester fund |
news <keyword> | News search | /data-harvester news AI |
compare <product> | Competitor comparison | /data-harvester compare laptop |
batch <file> | Batch URL list | /data-harvester batch urls.txt |
export <file> <format> | Export data | /data-harvester export data.json excel |
Usage Examples
Stock Query
/data-harvester stock 000001Returns: Real-time stock price, change %, volume
Competitor Comparison
/data-harvester compare "wireless earphones"Returns: Price comparison across Taobao, JD, Pinduoduo
Sector Analysis
/data-harvester blockReturns: Top 7hot sectors with leader stocks
Fund Flow
/data-harvester fundReturns: Top 5 stocks with main fund inflow
Batch Processing
Create a text file with one URL per line:
https://example1.com
https://example2.com
https://example3.comThen run:
/data-harvester batch urls.txtTechnical Details
- Language: Python 3
- Output Formats: JSON, CSV, Excel
- Rate Limiting: Configurable delay between requests
Legal Notice
- Respect target website's robots.txt
- Use reasonable request intervals (3-5 seconds)
- For educational/research purposes only
- Commercial use requires proper authorization