supercrawler

Version:

A web crawler. Supercrawler automatically crawls websites. Define custom handlers to parse content. Obeys robots.txt, rate limits and concurrency limits.

github.com/brendonboshell/supercrawler

brendonboshell/supercrawler

3 files

Filename	Content Type	Size
../
htmlLinkParser.spec.js	text/javascript	3.65 kB
robotsParser.spec.js	text/javascript	1.86 kB
sitemapsParser.spec.js	text/javascript	6.15 kB