Highly scalable crawler with best features.