UNPKG

robots-txt-parser

Version:

A lightweight robots.txt parser for Node.js with support for wildcards, caching and promises.

github.com/ChrisAkroyd/robots-txt-parser

ChrisAkroyd/robots-txt-parser

25 lines (22 loc) • 702 B
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
module.exports = ` User-agent: Googlebot Disallow: /iplayer/episode/*?from=r* Disallow: /iplayer/cy/episode/*?from=r* Disallow: /iplayer/gd/episode/*?from=r* Sitemap: http://www.bbc.co.uk/news_sitemap.xml Sitemap: http://www.bbc.co.uk/video_sitemap.xml Sitemap: http://www.bbc.co.uk/sitemap.xml User-agent: Googlebot-Mobile Sitemap: http://www.bbc.co.uk/mobile_sitemap.xml Disallow: /_programmes Disallow: /606/ Disallow: /aboutthebbc/insidethebbc/search Disallow: /academy/chinese-trad/search User-agent: * Sitemap: http://www.bbc.co.uk/sitemap.xml Disallow: /_programmes Disallow: /606/ Disallow: /aboutthebbc/insidethebbc/search Disallow: /academy/chinese-trad/search Disallow: /afrique/search `;