Wednesday 31 May 2017

How To Read Big Files in NodeJS?

I'm trying to read a 20 million lines file and correct the line endings from windows to mac. I know it can be done in sed but sed gives me an error that I don't know how to fix (dos2unix: Binary symbol 0x0008 found at line 625060). So I'm trying to fix this in NodeJS. Here's my code:

var fs = require('fs');
var eol = require('eol');

//read file
var input = fs.readFileSync(process.argv[2], 'utf8');

//fix lines
output = eol.auto(input);
console.log("Lines Fixed! Now Writing....")

//write file
fs.writeFile(process.argv[2] + '_fixed.txt', output, function (err) {
  if (err) return console.log(err); 
});
console.log("Done!")

Problem is the file is too big and I get this error buffer.js:513 throw new Error('"toString()" failed');



via Dany M

No comments:

Post a Comment