Wednesday, 15 March 2017

Node app that fetches, processes, and formats data for consumption by a frontend app on another server

I currently have a frontend-only app that fetches 5-6 different JSON feeds, grabs some necessary data from each of them, and then renders a page based on said data. I'd like to move the data fetching / processing part of the app to a server-side node application which outputs one simple JSON file which the frontend app can fetch and easily render.

There are two noteworthy complications for this project:

1) The new backend app will have to live on a different server that its frontend counterpart

2) Some of the feeds change fairly often, so I'll need the backend processing to constantly check for changes (every 5-10 seconds). Currently with the frontend-only app, the browser fetches the latest versions of the feeds on load. I'd like to replicate this behavior as closely as possible

My thought process for solving this took me in two directions:

The first is to setup an express application that uses setTimeout to constantly check for new data to process. This data is then sent as a response to a simple GET request:

const express = require('express');
let app = express();
let processedData = {};
const getData = () => {...} // returns a promise that fetches and processes data


/* use an immediately invoked function with setTimeout to fetch the data 
 * when the program starts and then once every 5 seconds after that */
(function refreshData() {
  getData.then((data) => {
    processedData = data;
  });

  setTimeout(refreshHomepage, 5000);
})();

app.get('/', (req, res) => {
  res.send(processedData);
});

app.listen(port, () => {
  console.log(`Started on port ${port}`);
});

I would then run a simple get request from the client (after properly adjusting CORS headers) to get the JSON object.

My questions about this approach are pretty generic: Is this even a good solution to this problem? Will this drive up hosting costs based on processing / client GET requests? Is setTimeout a good way to have a task run repeatedly on the server?

The other solution I'm considering would deal with setting up an AWS Lambda that writes the resulting JSON to an s3 bucket. It looks like the minimum interval for scheduling an AWS Lambda function is 1 minute, however. I imagine I could set up 3 or 4 identical Lambda functions and offset them by 10-15 seconds, however that seems so hacky that it makes me physically uncomfortable.

Any suggestions / pointers / solutions would be greatly appreciated. I am not yet a super experience backend developer, so please ELI5 wherever you deem fit.



via dougmacklin

No comments:

Post a Comment