A lightweight robots.txt parser for Node.js with support for wildcards, caching and promises.
github.com/ChrisAkroyd/robots-txt-parser
ChrisAkroyd/robots-txt-parser