@benchristel/supercrawler
Version:
A web crawler. Supercrawler automatically crawls websites. Define custom handlers to parse content. Obeys robots.txt, rate limits and concurrency limits.
| Filename | Content Type | Size | |
|---|---|---|---|
| ../ | |||
| 10.5 kB | |||
| 571 B | |||
| 20.9 kB | |||
| 11.2 kB | |||
| 3.09 kB | |||
| 1.99 kB | |||
| 1.35 kB | |||