Saturday, 27 May 2017

NodeJS read huge directory file by file

I have a directory composed of empty files and I want to iterate through each of their names. However I have nearly 20 million of these empty files and to load them all into memory with fs.readdir or fs.readdirSync would both take needlessly long and devour all my memory on the system.

What would be a way to go about this?

Ideally I would look for something that reads file by file in an async fashion with code that would ressemble the following:

readdirfilebyfile((filename)=>{....}) so that at no point would I keep the entire list of files in memory.

The current solution I am using is dumping all the file names into a single file which I then read as a data stream. However, this is just running away from a problem that I should know how to solve without resorting to this.



via Slava Knyazev

No comments:

Post a Comment