I am able to download very large ~2000 files from s3 bucket to local disk in python using boto.s3.resumable_download_handler and boto.s3.connection. download and archive to tar file in local disk. It works.
I am trying the same functionality in nodejs: If I am trying to download below ~500 files using s3 getObject and archive . Files gets downloaded and archived to local disk (using nodejs).
I am trying to download very large ~2000 files using s3 getObject and archive these files from the server to local disk using archive in nodejs. I am getting s3 error: Please reduce your request rate.ServiceUnavailable: Please reduce your request rate.
Code Snippet:
var AWS = require('aws-sdk');
var archiver = require('archiver');
var fs = require('fs');
var OutputFileName = 'abc.zip'
s3Client.listObjects(parameters, function(error, data) {
var dataConts = data.Contents;
dataConts.forEach(function(dataE) {
var aparams = {Bucket: bucketName, Key: dataE.Key};
archive.append(s3Client.getObject(aparams).createReadStream(),
{name: dataE.Key});
//archive on error
archive.on('error', function(error) {
throw error;
});
//archive on end of processing
archive.on('end', function() {
callback();
});
response.attachment(OutputFileName);
archive.pipe(respose);
archive.finalize();
can anyone let me know whether s3 and archive can be used in any other way to download and archive the very large number of files.
via Yamini
No comments:
Post a Comment