@benchristel/supercrawler

Version:

A web crawler. Supercrawler automatically crawls websites. Define custom handlers to parse content. Obeys robots.txt, rate limits and concurrency limits.

github.com/brendonboshell/supercrawler

benchristel/supercrawler

3 files

Filename	Content Type	Size
../
htmlLinkParser.spec.js	text/javascript	3.2 kB
robotsParser.spec.js	text/javascript	1.86 kB
sitemapsParser.spec.js	text/javascript	5.49 kB