UNPKG

spiking

Version:

A lightweight web crawler.

github.com/Xciyang/spiking

Xciyang/spiking

25 lines (13 loc) • 980 B
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
## Usage First select the crawl mode, `Multiple` for multi-line concurrent crawling, `DynamicMultiple` for dynamic web page concurrent crawling. `URL` Enter the link you want to crawl. `Local Path` Select local storage path. `Concurrency` Enter the maximum number of concurrency. (This option is related to the crawl speed and is limited by the performance of the computer.) `Proxy` Enter the proxy address. `Chormo Path` Select the Chormo browser executable file. `Display` Select whether to display the browser window. `Login` Select whether you want to log in (please select the display first). If 'Login' is selected, the program will enter the wait and click 'Continue' to begin crawling. Crawling is done automatically, and when everything is over, you can retry the failed link. ## More When you choose the `DynamicMultiple` mode, you not only need to enter the correct proxy address in the program, but also need to change the IE proxy.