simplecrawler
Version:
Very straightforward, event driven web crawler. Features a flexible queue interface and a basic cache mechanism with extensible backend.
3 lines (2 loc) • 46 B
JavaScript
Very straightforward, event driven web crawler. Features a flexible queue interface and a basic cache mechanism with extensible backend.