- name
- douyin-content-tracker
- description
- This skill should be used when the user wants to scrape Douyin (TikTok China) creator content, download audio, and transcribe it with Whisper. Covers first-time setup, daily incremental tracking, cookie refresh, and debugging. All pipeline scripts are bundled in this skill directory and can be run directly without any extra installation beyond pip and MediaCrawler.
Douyin Content Tracker
Scrapes Douyin creator videos via MediaCrawler, downloads audio with ffmpeg, and transcribes speech with Whisper.
Finding the Skill Base Directory
All commands must run from this skill's directory. To locate it, run:
python -c "import pathlib; print([p for p in pathlib.Path.home().rglob('douyin-content-tracker-skill/SKILL.md')])"Or check common locations:
~/.claude/skills/douyin-content-tracker-skill/- The path shown when the skill was installed
Set it as a variable for convenience:
SKILL_DIR="~/.claude/skills/douyin-content-tracker-skill" # adjust to actual path
cd "$SKILL_DIR"First-Time Setup
Run these steps once on a new machine.
1. Install Python dependencies
cd $SKILL_DIR
pip install -r scripts/requirements.txt
python -m playwright install chromium2. Install MediaCrawler
# Windows
git clone https://github.com/NanmiCoder/MediaCrawler D:/MediaCrawler
cd D:/MediaCrawler && pip install -r requirements.txt
# macOS/Linux
git clone https://github.com/NanmiCoder/MediaCrawler ~/MediaCrawler
cd ~/MediaCrawler && pip install -r requirements.txt3. Configure .env
cd $SKILL_DIR
cp .env.template .envEdit .env — required field:
MEDIACRAWLER_DIR=D:/MediaCrawler # adjust to actual MediaCrawler path (use ~/MediaCrawler on macOS/Linux)Optional overrides:
# Where to store data/audio/subtitles/models (default: ~/DouyinContentTracker or %USERPROFILE%\DouyinContentTracker)
OUTPUT_BASE_DIR=/Users/me/DouyinContentTracker
# Whisper model size (default: medium)
WHISPER_MODEL=small4. Add target accounts
Edit accounts.txt (or set TRACKER_ACCOUNTS_FILE / pass --accounts-file when running):
博主名称 | https://www.douyin.com/user/MS4wLjABAAAA...5. First login (generates cookie)
cd $SKILL_DIR
python scripts/scrape_profile.pyA browser opens — scan the Douyin QR code to log in. Cookie is saved to .douyin_cookies.json.
Daily Usage
cd $SKILL_DIR
# Track latest 3 videos per account (default). main.py mirrors track_latest.py
python scripts/track_latest.py
# or
python scripts/main.py
# Track latest N videos
python scripts/track_latest.py --limit 5
# Use a custom account list (also works via env TRACKER_ACCOUNTS_FILE)
python scripts/track_latest.py --accounts-file /path/to/accounts.txt
# Skip audio download and transcription (data only)
python scripts/track_latest.py --no-audioCookie Refresh
When scraping returns 0 videos or warns "Cookie 已 N 天未更新":
cd $SKILL_DIR
python scripts/scrape_profile.py # opens browser, scan QRPipeline Flow
accounts.txt (or the list pointed by --accounts-file / TRACKER_ACCOUNTS_FILE)
↓
scripts/scrape_profile.py → MediaCrawler (CDP) → OUTPUT_BASE_DIR/data/*.csv
↓
scripts/clean_data.py → normalized OUTPUT_BASE_DIR/data/cleaned_*.csv
↓
scripts/download_video.py → Playwright + ffmpeg → OUTPUT_BASE_DIR/audio/{blogger}/*.m4a
↓
scripts/extract_subtitle.py → Whisper → OUTPUT_BASE_DIR/subtitles/{blogger}/{video_id}.mdOutput Locations
All generated files live under OUTPUT_BASE_DIR (defaults to ~/DouyinContentTracker on macOS/Linux, %USERPROFILE%\DouyinContentTracker on Windows).
| Subdir | Contents |
|---|---|
data/cleaned_*.csv | Scraped + normalized video metadata |
audio/{blogger}/{video_id}.m4a | Extracted audio |
subtitles/{blogger}/{video_id}.md | Whisper transcript (title as first line) |
subtitles/{blogger}.md | All transcripts for one blogger merged |
Execution Logging Guide
When running the pipeline, report progress to the user after each step completes. Do not wait until the entire pipeline finishes.
Step-by-step reporting template:
After each Bash tool call returns, immediately tell the user:
| Step | What to report |
|---|---|
| 整理（scrape） | 博主名称、整理到的视频条数，若失败注明原因 |
| 清洗（clean） | 清洗后有效条数 |
| 音频下载（download） | 成功下载的音频数 / 总数，跳过的条数 |
| 语音识别（whisper） | 生成的字幕文件数，输出路径 |
| 完成 | 汇总：共处理博主数、视频数、生成字幕数，以及输出目录路径 |
If a step fails, stop the pipeline, report the error output verbatim, and suggest the matching fix from references/troubleshooting.md before asking the user whether to continue.
Example output style:
[步骤 1/4 整理] 博主「某某」— 整理完成，共 10 条视频
[步骤 2/4 清洗] 有效数据 10 条 → data/cleaned_profile_xxx.csv
[步骤 3/4 音频] 下载完成 8/10（2 条无音频流，已跳过）
[步骤 4/4 字幕] 生成 8 个字幕文件 → subtitles/某某/
[完成] 1 位博主 · 10 条视频 · 8 个字幕，输出目录：~/DouyinContentTrackerReferences
Load these files into context when debugging or extending the pipeline:
references/pipeline.md— per-script technical breakdown, data schemas, key function signaturesreferences/troubleshooting.md— fixes for cookie, MediaCrawler, ffmpeg, Whisper, and data errors