Question
In Node.js writable-stream, are there any way to call _write() in parallel?
What I hope & try (& fail)
I hope to execute asynchronous functions in parallel in writable._write with highWaterMark.
I want to upload many files (over 10,000 !!) to cloud in parallel.
Of course, 10,000 parallel upload cause memory overflow, so I plan like below.
"filename" Readable-stream => "upload" Writable-steream
[highWaterMark](parallel limit)
So I test like below.
class uploadWritable extends Writable {
constructor() {
const options = {
objectMode: true,
highWaterMark: 10
};
super(options);
}
_write(filepath, encoding, callback){
asyncUpload(filepath, callback); // When upload finished, callback is called.
}
}
I expect that
10 "asyncUpload" is called in parapell. And when one of "asyncUpload"s finish, new file is come from readable-stream and start "asyncUpload".
But above code cause "one by one" upload.
Question
Is above failure because of my code?
Are there any way to execute async function in parallel with Node.js writable-stream?
via pandaYK
No comments:
Post a Comment