UNPKG

robots-txt-parser

Version:

A lightweight robots.txt parser for Node.js with support for wildcards, caching and promises.

github.com/ChrisAkroyd/robots-txt-parser

ChrisAkroyd/robots-txt-parser

34 lines (30 loc) • 688 B
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
module.exports = ` # Comments should be ignored. # Short bot test part 1. User-agent: Longbot Allow: /cheese Allow: /swiss Allow: /swissywissy Disallow: /swissy Crawl-delay: 3 Sitemap: http://www.bbc.co.uk/news_sitemap.xml Sitemap: http://www.bbc.co.uk/video_sitemap.xml Sitemap: http://www.bbc.co.uk/sitemap.xml User-agent: MoreBot Allow: /test Allow: /special Disallow: /search Disallow: /news Crawl-delay: 89 Sitemap: http://www.bbc.co.uk/sitemap.xml User-agent: * Allow: /news Allow: /Testytest Allow: /Test/small-test Disallow: / Disallow: /spec Crawl-delay: 64 Sitemap: http://www.bbc.co.uk/mobile_sitemap.xml Sitemap: http://www.bbc.co.uk/test.xml host: http://www.bbc.co.uk `;