supercrawler

Version:

A web crawler. Supercrawler automatically crawls websites. Define custom handlers to parse content. Obeys robots.txt, rate limits and concurrency limits.

github.com/brendonboshell/supercrawler

brendonboshell/supercrawler

3 files

Filename	Content Type	Size
../
htmlLinkParser.js	text/javascript	1.38 kB
robotsParser.js	text/javascript	868 B
sitemapsParser.js	text/javascript	2.55 kB