Sunday, 12 March 2017

Node.js: 100M elements in 1D vs 2D Array — Unexpected Memory Issue

Let's say I have 100 million random floats I'm trying to save in an array.

At first, I was saving the numbers in a 2D array (basically a matrix), but then I thought maybe I could make the whole thing faster if I stored all 100M elements in a single linear 1D array.

To my surprise, this happened:

10 000 × 10 000 2D Array

Creation and manipulation of the array is quite fast, Node runs without any troubles even on default memory limit.

100M—element 1D Array

When trying to generate 100M random floats Node throws
FATAL ERROR: invalid array length Allocation failed - JavaScript heap out of memory
even when called with --max_old_space_size=8192

Why is that?

It seems very counter-intuitive to me. My guess was making a single long Array object should be much more memory-efficient than storing 10k Arrays in another Array.

So, yeah, my question is: why is that?



via R. Kysely

No comments:

Post a Comment