UNPKG

robots-txt-parser

Version:

A lightweight robots.txt parser for Node.js with support for wildcards, caching and promises.

github.com/ChrisAkroyd/robots-txt-parser

ChrisAkroyd/robots-txt-parser

36 lines (33 loc) • 822 B
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
module.exports = ` User-agent: * Disallow: /*?fh_location* Disallow: /search?* Disallow: /*?price* Disallow: /*?aspxerrorpath* Disallow: /*?product* Disallow: /*?tt* Disallow: /*?category* Disallow: /*?CID* Disallow: /*?nietgevonden* Disallow: /*?viewmode* Disallow: /*?cIdee* Disallow: /*?orderId* Disallow: /*?token* Disallow: /klantenservice?searchQuery=* Disallow: /productdetail* Disallow: /artikel=* #Blog User-agent: * Disallow: /blog/wp-admin/ Disallow: /blog/tag/ Disallow: /page/ Disallow: /blog/ultimate_slider* #Sitemap User-Agent: * Sitemap: https://www.bcc.nl/sitemap.xml Sitemap: https://www.bcc.nl/cms-sitemap.xml Sitemap: https://www.bcc.nl/video-sitemap.xml Sitemap: https://www.bcc.nl/blog/sitemap.xml Sitemap: https://www.bcc.nl/brands-sitemap.xml Sitemap: https://www.bcc.nl/products-sitemap.xml `;