- name
- crawler
- version
- 3.0.0
- description
- Web crawling and scraping reference — robots.txt protocol, Scrapy framework, anti-bot detection, headless browsers, and legal considerations
- author
- BytesAgain
- homepage
- https://bytesagain.com
- source
- https://github.com/bytesagain/ai-skills
- tags
- [web-scraping, scrapy, crawler, robots-txt, selenium]
- category
- devtools
Crawler
Web crawling and scraping reference — robots.txt protocol, Scrapy framework, anti-bot detection, headless browsers, and legal considerations. No API keys or credentials required — outputs reference documentation only.
Commands
| Command | Description |
|---|---|
intro | Crawling vs scraping, robots.txt, sitemap |
standards | HTTP caching, structured data, meta tags |
troubleshooting | Anti-bot detection, JS rendering, encoding |
performance | Concurrency, dedup, incremental, distributed |
security | Legal landscape, ethical guidelines, proxies |
migration | BeautifulSoup to Scrapy, requests to Playwright |
cheatsheet | Scrapy commands, CSS/XPath, curl, user-agents |
faq | Legality, JS pages, blocking, storage |
Output Format
All commands output plain-text reference documentation via heredoc. No external API calls, no credentials needed, no network access.
*Powered by BytesAgain | bytesagain.com | hello@bytesagain.com*