Thursday 20 April 2017

How do i allocate enough ram to run Object.stringify without a memory / RangeError

I'm sending a very large JSON file using a node instance.

The JSON file is about 150MB.

When i run it in production, i get the following error:

  RangeError: Invalid string length
      at join (native)
      at Object.stringify (native)
      at stringify (/opt/biz/fishco/api/node_modules/json-stringify-safe/stringify.js:5:15)
      at safeStringify (/opt/biz/fishco/api/node_modules/request/lib/helpers.js:24:11)
      at Request.json (/opt/biz/fishco/api/node_modules/request/request.js:1222:17)
      at Request.init (/opt/biz/fishco/api/node_modules/request/request.js:407:10)
      at Request.RP$initInterceptor [as init] (/opt/biz/fishco/api/node_modules/request-promise-core/configure/request2.js:45:29)
      at new Request (/opt/biz/fishco/api/node_modules/request/request.js:128:8)
      at request (/opt/biz/fishco/api/node_modules/request/index.js:54:10)
      at MachineModel.sendModelToML (/opt/biz/fishco/api/dist/src/biz/models/machine.model.js:28:16)
      at MachineController.<anonymous> (/opt/biz/fishco/api/dist/src/biz/controllers/machine.controller.js:44:35)
      at next (native)
      at fulfilled (/opt/biz/fishco/api/dist/src/biz/controllers/machine.controller.js:4:58)
      at process._tickCallback (internal/process/next_tick.js:103:7)

Some quick googeling said this might be a VM memory issue and to expand the HEAP size.

My server has 32GB of RAM. My node process is dockerized. Docker is supposed to share available memory. I've started the node process using nodemon I've set used npm start with the folliwing:

"scripts": {
    "start": "node --max_old_space_size=16384 --optimize_for_size --max_executable_size=16384 --stack_size=16384 dist/src/application",
}

This should be plenty of memory.

I've also created a local test harness that shows this utilizes about 1400MB of memory.

import fs = require("fs");
let request = require("request-promise");

console.log(__dirname);
let text = fs.readFileSync('/Users/jmurphy/projects/test_harness/target.json', 'utf8');
let json = JSON.parse(text);

let backToText = JSON.stringify(text);

let options: any = {
  uri: `http://localhost`,
  method: 'POST',
  json: json
};

console.log(request(options));

This runs correctly, and i track the memory usage in htop.

Question: How do i ensure there's enough memory available to properly stringify?



via Jack Murphy

No comments:

Post a Comment