Monday, 22 May 2017

Multi-process scraper in Nightmare.js

I'm developing a scraper in Nodejs using the npm package NightmareJS.

The scraper works perfectly if I run a single process. But when I run it in a multi-process way something very curious happen: Basically all the nightmare instances start successfully but then only one of them is executed while the others stop working.

I'm thinking that probably the last instance overrides the previous ones. Does anyone has any suggestion/idea?

To fork the process I use this code:

childVector[i] = cp.fork("./file.js");
childVector[i].send(JSON.stringify(informations));
childVector[i].on('message', function(message) {
       doStuff();                      
});



via Jack

No comments:

Post a Comment