Easily build flexible, scalable, and distributed, web crawlers.
github.com/eckardto/flexible
eckardto/flexible