UNPKG

robots-txt-parser

Version:

A lightweight robots.txt parser for Node.js with support for wildcards, caching and promises.

github.com/ChrisAkroyd/robots-txt-parser

ChrisAkroyd/robots-txt-parser

23 lines (22 loc) • 669 B
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
module.exports = ` User-agent: * Disallow: /aboutthebbc/insidethebbc/search Disallow: /academy/chinese-trad/search Disallow: /afrique/search Disallow: /apps/cbbc Disallow: /apps/flash Disallow: /apps/ide Disallow: /apps/ids Disallow: /apps/ifl Disallow: /apps/nr Disallow: /apps/query Disallow: /apps/t Disallow: /arabic/search Disallow: /arts/yourpaintings/artdetective/ Disallow: /arts/yourpaintings/feedback/sent/ Disallow: /arts/yourpaintings/images/guidedtours/ Disallow: /arts/yourpaintings/newsletter/error/ Disallow: /arts/yourpaintings/newsletter/success/ Disallow: /arts/yourpaintings/newsletter/unsubscribe/ Disallow: /arts/yourpaintings/paintings/search `;