Parse robot directives within HTML meta and/or HTTP headers.
github.com/stevenvachon/robot-directives
stevenvachon/robot-directives