Thursday, 18 May 2017

Pass large array to node child process

I have complex CPU intensive work I want to do on a large array. Ideally, I'd like to pass this to the child process.

    var spawn = require('child_process').spawn;

    // dataAsNumbers is a large 2D array
    var child = spawn(process.execPath, ['/child_process_scripts/getStatistics', dataAsNumbers]);

    child.stdout.on('data', function(data){
        console.log('from child: ', data.toString());
    });

But when I do, node gives the error:

spawn E2BIG

So it looks like I cannot pass large arguments to the script. Is this correct?

So instead, in my child process script, I decided to then fetch the data from Mongo. But when I require in my Mongoose schema:

var mongoose = require('mongoose');
var FileSchema = mongoose.model('FileSchema');

the child process doesn't execute at all and I cant see any errors.

What's the normal way of doing things when you need a child process to process a large amount of data that you can't pass to it?



via Mark

No comments:

Post a Comment