AWS Lambda Design Pattern

I have been playing with lambda functions to see if they fit my use case.

My lambda functions are just an abstraction of another API and aims to reduce the number of calls needed to set up an object on an external system. Because the http requests can take a good bit to process, and most of them have to be done synchronously, i'm running in to API gateway timeout errors with large concurrent connections. I know the API gateway timeout is set to 30 seconds and cannot be extended.

My current pattern takes in a large payload, breaks it down in to chunks, invokes a "worker" lambda and passes a chunk to it to do all of the http request in parallel. Once the workers are done the responses are passed back to the handler and the response is crafted and returned to the client.

This starts to fail when I have large concurrent requests to the handler (the api gateway times out).

I'm not necessarily looking to "save" my current code... I'd rather know the correct approach to scale and go that route.

Here's the code for the handler:

/*
*
*   Generic Handler
*     - Handler will parse JSON request and batch to appropriate worker
*     - Handler will receive finished batch jobs, combine, and return response. 
*
*/ 


const WORKER_LAMBDA_FUNCTION_NAME = "worker-name";
const REGION = "us-west";
const BATCH_SIZE = 50;

var AWS = require("aws-sdk");
var async = require("async");
var lambda = new AWS.Lambda({region: REGION});

/*
*   Batch objects in groups of BATCH_SIZE for workers
*/
const batchObjects = (allObjects) => {
  var currentBatch = [];
  var allBatch = [];
  //batch
  if (allObjects && allObjects.length > 0) {
    allObjects.forEach((s)=>{
      if(currentBatch.length <= BATCH_SIZE){
        currentBatch.push(s);
      } else {
        allBatch.push(currentBatch);
        currentBatch = [];
        currentBatch.push(s);
      }
    });
  }
  //include left overs
  if(currentBatch.length > 0){
    allBatch.push(currentBatch);
  }

  return allBatch;
};

/*
*   Invokes new Lambda function as worker.
*/

const invokeWorkerLambda = (task, callback) => {
  var payload ={};
  payload.MyObjects = task;
  payload.Authorization = API_KEY;

  var params = {
    FunctionName: WORKER_LAMBDA_FUNCTION_NAME,
    InvocationType: 'RequestResponse',
    Payload: JSON.stringify(payload)
  };

  lambda.invoke(params, (err, data) => {
    if (err) {
      console.error(err, err.stack);
      callback(err);
    } else {
      callback(null, data);
    }
  });
};

/*
*   Generic Handler...
*   Not a huge fan of this structure.... should clean this up eventually
*/
const handleGeneric = (req, callback) => {
  const allObjects = req.body.MyObjects;
  const batches = batchObjects(allObjects);
  var invocations = [];

  if(batches && batches.length > 0){
    batches.forEach((batch)=> {
      invocations.push((callback)=> {
        invokeWorkerLambda(batch, callback);
      });
    });

    async.parallel(invocations, (err, response) => {
      if (err) {
        console.error(err, err.stack);
        callback(err);
      } else {
        var returnValue = response[0];
        callback(null, JSON.parse(returnValue.Payload));
      }
    });
  } else {
    //no message found
    callback(null, "No Message Found");
  }
};

/*
*   Exported Lambda Method
*/
exports.test = function (event, context, callback) {
  console.log('Authorization ', event.headers.Authorization);
  if(event.headers.Authorization === undefined)
    return "No API Key Specified";

  API_KEY = event.headers.Authorization;

  handleGeneric(event, callback);
};

The workers just run http requests and return to the handler.

via Ross Bassett