Tuesday 23 May 2017

Zip Create Process with Node Express of large ZIP packages

Goal

We standing up a low volume site, where users will select map image files (284KB per file) and then request an Node Express Server to bundle them into a ZIP for download to the web client.

Issues & Design Constraints

  • The resultant ZIP might be on the order of 50MB - 5 GB. Therefore we would like to give the user a running progress bar while the ZIP is being constructed. (I assume the browser will give running updates as to the progress of the actual download.
  • While we expect low volume of requests (1-2 request at a time). However, we do not want to completely tie up our 4 core server processor, so we want async calls for the creation of the zip
  • Is there any other issues we should worry about?

Question

We assume that running 7zip as a child process is bad, since we would not get any running status as to how many of the 258KB files had been added to the ZIP.

So which of the following are most Node/ExpressJS friendly packages given the design constraints/goals listed above?

What I am seeing above is that some collect the files, and then finalize them to memory and then pipe them to the http request (probably not good for 5GB of data or am I missing something).

Others seem to be fully async and I don't see how you would get a running progress value as each file added to the ZIP package.



via Dr.YSG

No comments:

Post a Comment