Monday, 17 April 2017

How to resolve memory leak in node.js due to .map function

I am trying to export 9 million records from my mssql database to a mysql database using a node.js app

I am running into a memory issue which I suspect is due to my map functions.( Bluebird Promise.map)

The below Myfucntion() calls the doThis() function which gets a set of records from my Mssql database then insert the record into a MySql database.

Myfucntion(){
Promise.map(ids, id => doThis(id).then(results => console.log(results)), { concurrency: 5});
}

const doThis = async (id) => {
  try{
       const results = await sql.query`select results where id = ${id}`;
        if((results && results.recordset) && results.recordset.length > 0 ) {
          results.recordset.map(asset =>  insertResultslAsset(convertResultAsset(asset)));
        }
  }
  catch(err){
      console.log(err)
  }
 };

The convertResultAsset(asset) function creates a 4 properties object before doing the inserts.

I suspect the memory leakage occurs because the converted assets are being stored incrementally in memory with the .map iterations

I was able to alleviated the problem by running --max-old-space-size=20000 on the program.

After I perform the inserts from the doThis() function there is no need for me to store the converted Assets.

Is there a way to release these objects from the .map function?



via user1526912

No comments:

Post a Comment