I try to crawl a website using nightmarejs. I visit multiple pages and generate an array of data from the dom content for each of them. To do so, i return Q promises as in the nightmarejs documentation (github.com/segmentio/nightmare#promises)
I want to concatenate the arrays from all the crawled pages into a final array. To do so, i wait for all the promises returned by each crawling tasks to be fullfilled or rejected by using the allSettled function from the Q library (github.com/kriskowal/q/wiki/API-Reference#promiseallsettled)
The symptoms are : i can crawl one page returning a promise, wait it to resolve with allSettled, display results and so. But i can't crawl two pages returning promises, wait for them to resolve with allSettled, because the command line return the prompt without any errors or results. And i catch and display all results and errors all the way through the process...
Here is my cleaned code with comments : https://github.com/pierrediancourt/Template-NodeJsCrawler
Thanks for any insight you could bring to me. I can't figure out what is causing this weird behaviour, nightmarejs? allSettled? the particular combination of the two?
via pierrediancourt
No comments:
Post a Comment