robots-txt-parser

Version:

A lightweight robots.txt parser for Node.js with support for wildcards, caching and promises.

12 files

Filename	Content Type	Size
../
can-crawl-test-data.js	text/javascript	1.6 kB
example-parsed-robots-short.js	text/javascript	974 B
example-robots-malformed-short.js	text/javascript	468 B
example-robots-txt-allow.js	text/javascript	132 B
example-robots-txt-bbc.js	text/javascript	1.9 kB
example-robots-txt-bcc.js	text/javascript	822 B
example-robots-txt-blank-lines.js	text/javascript	405 B
example-robots-txt-disallow.js	text/javascript	669 B
example-robots-txt-karwei.js	text/javascript	798 B
example-robots-txt-short.js	text/javascript	688 B
example-robots-txt-sitemaps.js	text/javascript	702 B
example-robots-txt-zalando.js	text/javascript	506 B