Monday, 15 May 2017

AWS Lambda becomes unresponsive after TypeError (Node.js 4.3)

I'm using AWS Lambda with Node.js 4.3, and has DynamoDB get in the code:

dynamodb.get(params, function(err, data) {
    if (err) {
        console.log("Error: " + JSON.stringify(err));
        callback("Error: DB problem");
    }
    else {
            callback(null, data.Item.someAttribute);
    }
});

I know that it's wrong, and I need to check that Item actually exists before accessing it. That's not the point now.

I get TypeError: TypeError: Cannot read property 'someAttribute' of undefined

Which is ok, when there is no such item, it fails. The problem that after that lambda became to be unresponsive, there are other healthy requests come to that lambda, and lambda just fails every time with timeout. As you can see in the logs below (from CloudWatch) even 40(!) minutes after the error, the lambda is still unresponsive. And I know nothing happens there, as in the beginning if the code I'm printing "Input to Lambda" which can be seen in first request. In all times out requests there is no such print, Lambda just became dead.

START RequestId: xxx Version: 1
2017-05-12T09:20:18.739Z    Input to Lambda: ...
2017-05-12T09:20:19.770Z    xxx TypeError: Cannot read property 'someAttribute' of undefined at ...
END RequestId: xxx
REPORT RequestId: xxx   Duration: 1094.61 ms    Billed Duration: 1100 ms
RequestId: xxx Process exited before completing request

START RequestId: xxx Version: 1
END RequestId: xxx
REPORT RequestId: xxx   Duration: 3002.12 ms    Billed Duration: 3000 ms
2017-05-12T09:24:08.556Z xxx Task timed out after 3.00 seconds

...

START RequestId: xxx Version: 1
END RequestId: xxx
REPORT RequestId: xxx   Duration: 3001.15 ms    Billed Duration: 3000 ms
2017-05-12T10:01:14.672Z xxx Task timed out after 3.00 seconds

Fixing the code and reuploading it to the Lambda makes it working, but how can this be that some error make the whole endpoint unresponsive. Waiting some hours also makes Lambda work again. I'm thinking of removing "use strict", but it doesn't look like a good solution. Any thoughts on what can be the reason to dead lambda and how to avoid this in future?



via Slava Elantsev

No comments:

Post a Comment