Monday, 3 April 2017

How to resolve array join issue when you have huge object?

I have multiple lines event so make them single line event i have below logic but when we join arrays and push value to results it throw below error, Any idea how can i resolve this issue or proper solution join to objects of array ?

ctrl.js

 fs.readdir(dir,function(err, files){
            if (err) { throw err; }
               logFile = getNewestFile(files, dir)
               console.log(logFile);
            var items = [];
            var item = [];
            var readStream = fs.createReadStream(dir + '/' + logFile,'utf8');
                  readStream.pipe(split()).on('data', function (line) {
                      if (regex.test(line)) {
                          item = [];
                          items.push(item);
                      }
                      item.push(line);
                    var lines = items.map(function (item) {
                        return item.join(' ');
                    });
                    lines.forEach(function(line){
                      results.push({value:line});
                    });
                  });
              readStream.on('end',function(){
                var slicedArray = results.slice(Math.max(results.length - 200, 1));
                cb(null,slicedArray);
                results = [];
              });
        });

Error

 Last few GCs --->

    9751 ms: Mark-sweep 1367.6 (1457.6) -> 1367.6 (1457.6) MB, 106.3 / 0 ms [last resort gc].
    9752 ms: Scavenge 1367.6 (1457.6) -> 1367.6 (1457.6) MB, 0.6 / 0 ms [allocation failure].
    9752 ms: Scavenge 1367.6 (1457.6) -> 1367.6 (1457.6) MB, 0.4 / 0 ms [allocation failure].
    9850 ms: Mark-sweep 1367.6 (1457.6) -> 1367.6 (1457.6) MB, 97.6 / 0 ms [last resort gc].
    9948 ms: Mark-sweep 1367.6 (1457.6) -> 1367.6 (1457.6) MB, 98.0 / 0 ms [last resort gc].


<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0000027B9FCB4639 <JS Object>
    1: Join(aka Join) [native array.js:154] [pc=0000003E0961CF98] (this=0000027B9FC041B9 <undefined>,o=0000028FAF09D451 <JS Array[2]>,v=2,C=0000027B9FC908A9 <String[1]:  >,B=0000027B9FCB3E81 <JS Function ConvertToString (SharedFunctionInfo 0000027B9FC51C31)>)
    2: InnerArrayJoin(aka InnerArrayJoin) [native array.js:331] [pc=0000003E0961BE0A] (this=0000027B9FC041B9 <undefined>,C=0000027B9FC9...

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - process out of memory



via hussain

No comments:

Post a Comment