@benchristel/supercrawler

Version:

A web crawler. Supercrawler automatically crawls websites. Define custom handlers to parse content. Obeys robots.txt, rate limits and concurrency limits.

github.com/brendonboshell/supercrawler

benchristel/supercrawler

5 lines (4 loc) • 45 B

YAML

View Raw

language: node_js node_js: - "6" - "4.4"