Friday 2 June 2017

Node.JS: Why do nested promise thens seem to run out of order?

I need each download and getFiles call to run separately because firebase seems unable to handle too many connections.

const bucket = storage.bucket(bucketName); // firebase storage bucket
var queue = Promise.resolve();

websiteNames.reduce(function(promise, websiteName) {
    return promise
    .then(() => {
        mkdirp(websiteName)
    })
    .then(() => {
        console.log("now getting files: " + dirPath);
        return bucket.getFiles({prefix: dirPath})
    })
    .then(data => {
        console.log("got files");
        var files = data[0];
        files.forEach(function (file) {
            var storePath = tmpDir + "/" + file.name;
            queue = queue
            .then(() => {
                console.log("now downloading: " + file.name);
                return file.download({destination: storePath});
            })
            .then(() => {
                console.log(`downloaded to ${storePath}.`);
                return readFile(storePath, 'utf8');
            })
            .then(buffer => {
                console.log("readed file: " + storePath + " of buf size " + buffer.length);
            });
        });
    });
}, queue);

I expected that the big then block will be adding download actions to a queue which already has all the getFiles actions. So I expect it to make all the getFiles calls, before starting to download the files. But console logs like this:

now getting files: foo/a
got files
now downloading: foo/a/0.csv
now getting files: foo/b
got files
now getting files: foo/c
got files
downloaded to /tmp/foo/a/0.csv.

Why does foo/a/0.csv start downloading before all the bucket getFiles calls are made? Shouldn't the file download thens have been added to the end of the queue after it already had the getFiles thens? Moreover foo/a has multiple files, so why did only 1 file download sneak in between the getFiles calls?



via pete

No comments:

Post a Comment