Thursday, 27 April 2017

node.js cluster, redirecting the childs stdout to file breaks data

The task is:

  1. Cluster creates workers
  2. Each worker loads some remote data line by line and output them to its own process.stdout
  3. The output of the hole cluster is redirected to file

Expected result: all lines from all workers are concatenated to resulting file

Actual result: on some machines the result is such as expected, on others the result is interlaced.

Tested on 0.10.47 and 0.12.7 nodes under Ubuntu OS. On two machines under Ubuntu 16.04.2 the result is good. On one under Ubuntu 14.04.5 is interlaced.

I need help to understand this situation and fix it on 14.04.

cluster.js:

var cluster = require("cluster");

cluster.setupMaster({
    exec: "./worker.js"
});

cluster.fork();
cluster.fork();

worker.js:

var cluster = require("cluster");
var fs = require("fs");

var workerId = cluster.worker && cluster.worker.id || 1;
var stream = fs.createReadStream("./data" + workerId + ".log");

stream.on("end", function () {
    process.exit();
});

stream.pipe(process.stdout);

You need two files in the same folder named data1.log and data2.log with data to read:

$ wc -l data*
     9260 data1.log
   111636 data2.log
   120896 total

The piping always gives good result:

$ node cluster.js | wc -l
120896

The redirection to file gives good result only on Ubuntu 16.04 but never on Ubuntu 14.04. Even more on Ubuntu 14.04 it gives every time different results:

$ node cluster.js >result.log; wc -l result.log
114135      <= MUST BE 120896
$ node cluster.js >result.log; wc -l result.log
110136      <= MUST BE 120896

The redirection to file for addition always gives good result:

$ rm result.log
$ node cluster.js >>result.log; wc -l result.log
120896



via TheRoSS

No comments:

Post a Comment