robots-txt-parser
Version:
A lightweight robots.txt parser for Node.js with support for wildcards, caching and promises.
2 lines (1 loc) • 798 B
JavaScript
module.exports = 'User-agent: * Sitemap: https://sitemap.karwei.nl/index.xml Disallow: /formulierOptout-g Disallow: /*fh_sort_by=* Disallow: /*sort=* Disallow: /*fh_location* Disallow: /*zoeken* Disallow: /*locale=* Disallow: /*brand_name* Disallow: /*star_rating* Disallow: /*Fthema* Disallow: /*Dthema* Disallow: /*kleurfamilie* Disallow: /*amount* Disallow: /*verfbasis* Disallow: /*glansgraad* Disallow: /*Fprice* Disallow: /*Cprice* Disallow: /*Dprice* Disallow: /*2price* Disallow: /*rendement* Disallow: /*type_* Disallow: /*Fafwerking* Disallow: /*Dafwerking* Disallow: /*functional* Disallow: /*aantal* Disallow: /*net_width* Disallow: /*net_height* Disallow: /*?*f_* Disallow: /*?*fi_* Disallow: /favorieten Disallow: /favoris Disallow: /.pdf Disallow: /mooilaag/*$ Disallow: /*?*ean=*';