Saturday 27 May 2017

Nodejs: s3 getObject for very large number of files download and archive fails

I am able to download very large ~2000 files from s3 bucket to local disk in python using boto.s3.resumable_download_handler and boto.s3.connection. download and archive to tar file in local disk. It works.

I am trying the same functionality in nodejs: If I am trying to download below ~500 files using s3 getObject and archive . Files gets downloaded and archived to local disk (using nodejs).

I am trying to download very large ~2000 files using s3 getObject and archive these files from the server to local disk using archive in nodejs. I am getting s3 error: Please reduce your request rate.ServiceUnavailable: Please reduce your request rate.

   Code Snippet:
        var AWS = require('aws-sdk');
        var archiver = require('archiver');
        var fs = require('fs');
        var OutputFileName = 'abc.zip'

         s3Client.listObjects(parameters, function(error, data) {
var dataConts = data.Contents;
dataConts.forEach(function(dataE) {
    var aparams = {Bucket: bucketName, Key: dataE.Key};

                    archive.append(s3Client.getObject(aparams).createReadStream(),
                                            {name: dataE.Key});

           //archive on error
            archive.on('error', function(error) {
                throw error;
            });
            //archive on end of processing
            archive.on('end', function() {

                callback();
            });
            response.attachment(OutputFileName);

           archive.pipe(respose);
           archive.finalize();

can anyone let me know whether s3 and archive can be used in any other way to download and archive the very large number of files.



via Yamini

No comments:

Post a Comment