Robots parser that is used in our SEO application.
github.com/carbondigitalus/semlinks-robots-parser
carbondigitalus/semlinks-robots-parser