I am trying to export 9 million records from my mssql database to a mysql database using a node.js app
I am running into a memory issue which I suspect is due to my map functions.( Bluebird Promise.map)
The below Myfucntion() calls the doThis() function which gets a set of records from my Mssql database then insert the record into a MySql database.
Myfucntion(){
Promise.map(ids, id => doThis(id).then(results => console.log(results)), { concurrency: 5});
}
const doThis = async (id) => {
try{
const results = await sql.query`select results where id = ${id}`;
if((results && results.recordset) && results.recordset.length > 0 ) {
results.recordset.map(asset => insertResultslAsset(convertResultAsset(asset)));
}
}
catch(err){
console.log(err)
}
};
The convertResultAsset(asset) function creates a 4 properties object before doing the inserts.
I suspect the memory leakage occurs because the converted assets are being stored incrementally in memory with the .map iterations
I was able to alleviated the problem by running --max-old-space-size=20000 on the program.
After I perform the inserts from the doThis() function there is no need for me to store the converted Assets.
Is there a way to release these objects from the .map function?
via user1526912
No comments:
Post a Comment