Web Scraping Skill
This skill enables web scraping and data extraction capabilities.
When to Use
Use this skill when you need to:
- Scrape data from websites
- Interact with REST APIs
- Parse HTML content
- Export scraped data to various formats
Scraping Methods
Static HTML Scraping
For static websites, use CSS selectors to extract data.
Dynamic Content
For JavaScript-heavy sites, use browser automation.
API Access
For structured data access, prefer REST or GraphQL APIs.
Code Tools
This skill includes:
- @code: python scraper.py --input {input} - Generic web scraper
- @code: bash fetch.sh {url} - Simple HTTP fetcher
Best Practices
- Check robots.txt: Always respect site scraping policies
- Rate limiting: Add delays between requests to avoid overwhelming servers
- User-Agent: Identify your bot properly
- Error handling: Gracefully handle network errors and timeouts
Example
To scrape a webpage:
Use the web_scraping skill to extract product information from https://example.com/products