1.0.3 • Published 7 years ago

scrape-query-processor v1.0.3

Weekly downloads
1
License
ISC
Repository
-
Last release
7 years ago

#Scraping query processor

allows to scrape child urls from the site of the vendor with specified concurrency

data can be returned as an array of pages or saved as a bunch of files saveFiles flag

class accepts object parameters with following keys

- `dirPath` folder where all the files will be saved 

- `vendor`  will be used as a sub folder         
- `homeUrl` url of the main site, i.e `https://www.yamaha-motor.eu`

- `saveFiles` flag which allows saving of scraped pages

- `concurrency` amount of pages queried in parallel, default = 2

new QueryProcessor({vendor: 'yamaha`, homeUrl: 'https://www.yamaha-motor.eu', saveFiles: true, concurrency:5}