norobots
Version:
Robots.txt rules
15 lines (10 loc) • 346 B
Markdown
An attempt to implement the 1997 Internet Draft specification [A Method for Web Robots Control](http://www.robotstxt.org/norobots-rfc.txt).
```js
const norobots = require('norobots')
let robots = norobots('User-Agent: *\nDisallow: /cgi-bin/\n')
if (robots.allow('fig-tree', '/example.html')) {
// Go fetch the url
}
```