website-scrap-engine/README.md

Configurable website scraper in typescript

# website-scrap-engine
Configurable website scraper in typescript.

## Features
- [x] Resource types
- [x] Configurable process pipeline
- [x] Options
- [x] Logger
- [x] Concurrent downloader
- [x] Multi-thread processing (with native worker_thread)
- [x] Process CSS
- [x] Process HTML
- [x] Process SiteMap (but not replace path in it)
- [x] Configurable logging

## Multi-thread processing
Note: use multi-thread processing
only if your process is cpu sensitive.

* Main thread
    * resource downloading in queue
    * process after download
    * save binary resources to disk
    * send other resources to worker thread
    * enqueue non-duplicated resource from worker thread
* Worker thread
    * receive downloaded resource from main thread
    * process after download
        * parse html, css, etc.
    * collect referenced resources
    * process and filter referenced resources before download
    * send referenced resources to main thread
    * save resources to disk

## Pipeline life cycle
* skip or redirect link
* detect resource type
* create
* process before download
* download
* process after download
* save to disk