Sunday 21 May 2017

NodeJS on Lambda Using S3.getObject

So, I'm banging my head against the wall trying to figure out what in the world I am doing wrong here. I'm writing a Lambda function using NodeJS 6.10. The function is supposed to retrieve several images from one of my S3 buckets and process them. I am embracing the asynchronous nature of JavaScript, or trying to, anyway, and allowing these to happen concurrently and then wait and do something after all of them have completed. I've distilled the code here down to the bare minimum I can write and still reproduce the problem I'm seeing. The problem is that none of the images are ever retrieved - in fact, there's no indication that the Lambda function ever actually tries to retrieve them, and no indication that the callbacks are ever fired.

Here's the code:

console.log('Executing testImg method.');
var aws = require('aws-sdk');
var s3 = new aws.S3();

var imgData = {};

exports.handler = function(event, context, callback) {

    var imgs = ['img_1', 'img_2', 'img_3', 'img_4', 'img_5', 'img_6', 'img_7', 'img_8'];
    var imgPromises = [];
    console.log('Grabbing images: ' + imgs);
    imgs.forEach(function(img) {
        console.log(img);
        var myPromise = s3.getObject({
            Bucket: 'photoBucket',
            Key: 'images/' + img + '.png'
        }).promise();
        imgPromises.push(myPromise);
        myPromise.then(function(result) {
            console.log('Got ' + img);
            imgData[img] = result.Body;
        });
    });
    Promise.all(imgPromises).then(function() {
        console.log(imgData);
        context.succeed();
    });

}

And here is the output from Lambda:

START RequestId: <UUID> Version: $LATEST
2017-05-21T18:53:31.187Z    <UUID>  Grabbing images: img_1,img_2,img_3,img_4,img_5,img_6,img_7,img_8
2017-05-21T18:53:31.187Z    <UUID>  img_1
2017-05-21T18:53:31.625Z    <UUID>  img_2
2017-05-21T18:53:31.648Z    <UUID>  img_3
2017-05-21T18:53:31.706Z    <UUID>  img_4
2017-05-21T18:53:31.707Z    <UUID>  img_5
2017-05-21T18:53:31.708Z    <UUID>  img_6
2017-05-21T18:53:31.766Z    <UUID>  img_7
2017-05-21T18:53:31.767Z    <UUID>  img_8
END RequestId: <UUID>
REPORT RequestId: <UUID>    Duration: 10002.24 ms   Billed Duration: 10000 ms   Memory Size: 128 MB Max Memory Used: 80 MB  
2017-05-21T18:53:41.173Z <UUID> Task timed out after 10.00 seconds

As you can see from the output, the console.log(img) line gets run where it prints out the name of the image. However, none of the code inside the individual .then() blocks gets run (e.g. console.log('Got ' + img)), and none of the code in the final Promise.all block gets run (printing out the entire array, and calling the succeed function to terminate the function.

Also, a few other things I have tried:

  • Run the code locally using NodeJS + lambda-local. It works fine.
  • Run similar code in NodeJS without lambda-local. It works fine.
  • Run the code on Lambda using explicit credentials rather then allowing it to use a Lambda role. It still fails on AWS Lambda.
  • Downloading a single image works perfectly fine in Lambda - it's something about calling getObject multiple times that it doesn't seem to like, though I cannot find any indication in the documentation that this is a bad thing.
  • I have tried removing the individual myPromise.then() calls from inside the loop, and doing everything in the Promises.all() block, and, again, that does not result in the code actually running in AWS.

I could chain them all together so that they each happen synchronously, but 1) it's hard to do this in a dynamic fashion (not knowing the number of files I'm going to need to get ahead of time), and 2) this seems to go against everything that Async JS is about these days.

Any hints would be greatly appreciated.



via Virtually Nick

No comments:

Post a Comment