scrapping（scrapping 视频）

ScrapeCreators API

ScrapeCreators provides 100+ REST endpoints to scrape public data from 20+ social media platforms. One API key, one header, simple curl requests.

Quick start

Authentication

Every request needs a single header:

x-api-key: YOUR_API_KEY

Get your key at https://scrapecreators.com (100 free credits on signup, no card required).

Store the key in an environment variable so it stays out of scripts and doesn't end up in version control or chat history:

export SCRAPECREATORS_API_KEY="your-key-here"

Your first request

curl -s -G "https://api.scrapecreators.com/v1/tiktok/profile" \
  --data-urlencode "handle=khaby.lame" \
  -H "x-api-key: $SCRAPECREATORS_API_KEY" | jq .

That's the whole pattern. Every endpoint works the same way: GET request, query parameters, one auth header.

Base URL

https://api.scrapecreators.com

How credits work

• Most endpoints: 1 request = 1 credit
• A few specialized endpoints cost more (e.g., audience demographics = 26 credits)
• Credits never expire, no monthly commitment
• Most responses include a credits_remaining field
• Check credit costs per endpoint in the docs: https://docs.scrapecreators.com

Common parameters

These work across most endpoints:

Pagination

Many endpoints return paginated results. The pattern is:

1. Make the first request without a cursor
2. The response includes a cursor (or next_cursor or next_page_id) field
3. Pass that value as ?cursor=... in the next request
4. Repeat until cursor is null/empty

# First page
curl -s -G "https://api.scrapecreators.com/v3/tiktok/profile/videos" \
  --data-urlencode "handle=khaby.lame" \
  -H "x-api-key: $SCRAPECREATORS_API_KEY" | jq .

# Next page (using max_cursor from previous response)
curl -s -G "https://api.scrapecreators.com/v3/tiktok/profile/videos" \
  --data-urlencode "handle=khaby.lame" \
  --data-urlencode "max_cursor=CURSOR_VALUE" \
  -H "x-api-key: $SCRAPECREATORS_API_KEY" | jq .

Some v2 endpoints require manual pagination — check the platform reference for specifics.

Supported platforms and endpoints

Each platform has its own reference file with full endpoint details. Read the relevant one based on the user's request.

Making requests: the pattern

Every call follows the same shape. Always use -G with --data-urlencode to safely pass query parameters — this prevents shell injection from user-provided values (handles, search queries, URLs) and properly encodes special characters:

curl -s -G "https://api.scrapecreators.com/v1/{platform}/{endpoint}" \
  --data-urlencode "param=value" \
  -H "x-api-key: $SCRAPECREATORS_API_KEY"

Pipe through jq to format the JSON, or through jq '.some.field' to extract specific data. Using jq is important because raw API responses are often large nested JSON — extracting just the fields the user needs makes the output readable and keeps your context window clean.

Chaining requests

A common pattern is to fetch a profile first (to confirm the account exists and get IDs), then drill into their content:

# 1. Get profile
PROFILE=$(curl -s -G "https://api.scrapecreators.com/v1/tiktok/profile" \
  --data-urlencode "handle=creator_name" \
  -H "x-api-key: $SCRAPECREATORS_API_KEY")

# 2. Get their recent posts
POSTS=$(curl -s -G "https://api.scrapecreators.com/v3/tiktok/profile/videos" \
  --data-urlencode "handle=creator_name" \
  -H "x-api-key: $SCRAPECREATORS_API_KEY")

# 3. Get comments on a specific video (extract video ID from posts)
VIDEO_ID=$(echo "$POSTS" | jq -r '.aweme_list[0].aweme_id')
COMMENTS=$(curl -s -G "https://api.scrapecreators.com/v1/tiktok/video/comments" \
  --data-urlencode "video_id=$VIDEO_ID" \
  -H "x-api-key: $SCRAPECREATORS_API_KEY")

Saving results

# Save raw JSON
curl -s -G "https://api.scrapecreators.com/v1/instagram/profile" \
  --data-urlencode "handle=natgeo" \
  -H "x-api-key: $SCRAPECREATORS_API_KEY" > natgeo_profile.json

# Save as CSV (extract specific fields with jq)
curl -s -G "https://api.scrapecreators.com/v3/tiktok/profile/videos" \
  --data-urlencode "handle=creator" \
  -H "x-api-key: $SCRAPECREATORS_API_KEY" \
  | jq -r '.aweme_list[] | [.aweme_id, .desc, .statistics.play_count, .statistics.digg_count] | @csv' \
  > creator_posts.csv

Processing and analyzing scraped data

After fetching data, here are common analysis patterns:

Extract key metrics

# Get engagement stats from a profile
curl -s -G "https://api.scrapecreators.com/v1/tiktok/profile" \
  --data-urlencode "handle=creator" \
  -H "x-api-key: $SCRAPECREATORS_API_KEY" \
  | jq '{followers: .statsV2.followerCount, following: .statsV2.followingCount, likes: .statsV2.heartCount}'

Aggregate across posts

# Average engagement across recent posts
curl -s -G "https://api.scrapecreators.com/v3/tiktok/profile/videos" \
  --data-urlencode "handle=creator" \
  -H "x-api-key: $SCRAPECREATORS_API_KEY" \
  | jq '[.aweme_list[] | .statistics.digg_count] | (add / length)'

Get video transcripts for content analysis

# Fetch transcript of a TikTok video
curl -s -G "https://api.scrapecreators.com/v1/tiktok/video/transcript" \
  --data-urlencode "video_id=VIDEO_ID" \
  -H "x-api-key: $SCRAPECREATORS_API_KEY" | jq -r '.data.transcript'

Search and filter

# Find TikTok creators by niche with audience filters
curl -s -G "https://api.scrapecreators.com/v1/tiktok/creators/popular" \
  --data-urlencode "min_followers=100000" \
  --data-urlencode "audience_country=US" \
  -H "x-api-key: $SCRAPECREATORS_API_KEY" | jq '.data'

Important notes

• Public data only — the API scrapes publicly available information. No private accounts or DMs.
• No rate limits — run as many concurrent requests as needed.
• 29-second timeout — requests that take longer will time out (AWS Lambda limit). Most complete in ~3 seconds.
• AI transcript limit — videos over 2 minutes that need AI-generated transcripts won't return transcripts. YouTube transcripts have their own dedicated endpoint and are unaffected.
• Versioned endpoints — some endpoints have moved beyond v1: TikTok profile videos is now v3 (uses max_cursor instead of cursor), TikTok video info is v2, Instagram user posts / reels search / post comments are v2. Check the platform reference files for current versions.
• Response format — all responses are JSON. Use trim=true to reduce payload size when you only need key fields.

Error handling

If a request fails:

1. Check the HTTP status code
2. The response body usually contains an error message
3. Common issues: invalid API key (401), endpoint not found (404), timeout on slow requests (502/503)

# Check status code
curl -s -G -o response.json -w "%{http_code}" \
  "https://api.scrapecreators.com/v1/tiktok/profile" \
  --data-urlencode "handle=creator" \
  -H "x-api-key: $SCRAPECREATORS_API_KEY"

Choosing the right endpoint

When the user describes what they need, match it to the right platform and endpoint:

• "Get info about a creator/account" → /{platform}/profile
• "Get their posts/videos/content" → /{platform}/profile/videos (TikTok) or /{platform}/channel-videos (YouTube) or /{platform}/user/posts (Instagram) or /{platform}/posts (Facebook)
• "Get comments on a post" → /{platform}/video/comments or /{platform}/post/comments
• "Search for content about X" → /{platform}/search/keyword (TikTok) or /{platform}/search (others)
• "Get the transcript of a video" → /{platform}/video/transcript (TikTok/YouTube) or /v2/instagram/media/transcript (Instagram) or /facebook/transcript (Facebook)
• "Who follows them / who do they follow" → /{platform}/user/followers or /following
• "What's trending" → /tiktok/videos/popular, /tiktok/get-trending-feed, /tiktok/hashtags/popular
• "Find ads by a company" → /facebook/adLibrary/search/ads, /linkedin/ads/search, /google/company/ads, or /reddit/ads/search
• "Get product details from TikTok Shop" → /tiktok/product, /tiktok/shop/search

When in doubt, check the platform reference file for the full endpoint list.

Full API docs

For the complete interactive documentation with response schemas and playground: https://docs.scrapecreators.com