UNPKG

supercrawler

Version:

A web crawler. Supercrawler automatically crawls websites. Define custom handlers to parse content. Obeys robots.txt, rate limits and concurrency limits.

github.com/brendonboshell/supercrawler

brendonboshell/supercrawler

9 lines (7 loc) • 264 B
1
2
3
4
5
6
7
8
9
var utils = require("./utils"); module.exports = { RobotsNotAllowedError: utils.makeError("RobotsNotAllowedError"), HttpError: utils.makeError("HttpError"), RequestError: utils.makeError("RequestError"), HandlersError: utils.makeError("HandlersError") };