Communityv3.0.0

Crawler

Web crawling and scraping reference — robots.txt protocol, Scrapy framework, anti-bot detection, headless browsers, and legal considerations

1.5kdownloads6active installsbytesagain3
View on ClawHubBack to Skills

Skill Details

Slug
crawler
Latest Version
3.0.0
Author
bytesagain3
Published
Mar 15, 2026
Updated
May 11, 2026
Total Versions
10

How to Install

  1. 1 on OpenClawdBots (takes under 60 seconds).
  2. 2Open your bot dashboard and go to the Skills tab.
  3. 3Switch to the ClawHub tab and search for “Crawler”.
  4. 4Click Install and the skill is deployed to your bot automatically.

Changelog — v3.0.0

Clean package with matching SKILL.md and script