UNPKG

robots-txt-parser

Version:

A lightweight robots.txt parser for Node.js with support for wildcards, caching and promises.

77 lines (54 loc) 2 kB
# Changelog All notable changes to this project will be documented in this file. ## 2.0.0 - 2.0.0 release (21/06/2021) ### Changed - Rewrite of robots.js - Additional testing. - Fixed bugs with callbacks. ## 1.0.10 - Testing + cache usage (02/06/2021) ### Changed - Fixed lack of cache usage. - Additional testing. ## 1.0.9 - ESLint usage + related cleanup (23/05/2021) ### Changed - Fixed Coveralls reporting. - Fixed lcov output. ## 1.0.8 - ESLint usage + related cleanup (21/05/2021) ### Changed - Added eslint + better code style to old project. ## 1.0.6 - Documentation cleanup pass (16/05/2021) ### Changed - Fixed scroll-to links to work on both github + npm. ## 1.0.5 - Documentation cleanup pass (16/05/2021) ### Changed - Cleaned up readme file. - Documentation format improvements. - Fixed bad example. ## [1.0.4] - Minor additions and fixes (13/5/2021) ### Added - Additional testing on fetch command. - ClearCache() function. - Testing for clearCache(). ### Changed - Fixed no timeout on bad connection bug. - Fixed bad coverage test command. - Fixed error on connection to unavailable site. ### Removed - Unnecessary dependency. - Unnecessary test files. ## [1.0.3] - Minor additions and fixes (10/4/2021) ### Added - Parsing for Host records. ### Changed - Fixed bug with parsing robots.txt files stripped of line endings. - Fixed no user agent lower casing. - Fixed parser rule overwriting. - Fixed parser fallback. ## [1.0.0] - Initial release (1-05-2017) ### Added - Robots.txt parser. - Automated fetching. [Unreleased]: https://github.com/chrisakroyd/robots-txt-parser [1.0.4]: https://github.com/chrisakroyd/robots-txt-parser/tree/c549401d6f035b0e05b3045cca0ad5f2f51811ff [1.0.3]: https://github.com/chrisakroyd/robots-txt-parser/tree/f834932b97b667315ff28a9cf61a2f6c30d7e6d9 [1.0.0]: https://github.com/chrisakroyd/robots-txt-parser/tree/c062d954f834ca6b54729e607ba8f674ea01c919