UNPKG

robots-txt-parser

Version:

A lightweight robots.txt parser for Node.js with support for wildcards, caching and promises.

github.com/ChrisAkroyd/robots-txt-parser

ChrisAkroyd/robots-txt-parser

10 lines (9 loc) • 132 B
1
2
3
4
5
6
7
8
9
10
module.exports = ` User-agent: * Allow: /travel Allow: /weatherwatchers Allow: /music/ Allow: /news/ Allow: /test/ Allow: /harp/ `;