Thursday 4 May 2017

Getting ENOBUFS in a loop of http requests node js

Some background:
What I do is receive a CSV file with some ids from the client. I run on the file and for each ID I make need to make a rest query that is followed by another rest query.

This thing works fine when I do it for 1,000 to ~ 5,000 ids but after that most of my responses at the logs are errors instead of "200".

The error is:

[Error: connect ENOBUFS XX.XX.XX.XXX:8080 - Local (undefined:undefined)]
  code: 'ENOBUFS',
  errno: 'ENOBUFS',
  syscall: 'connect',
  address: 'XX.XX.XX.XXX',
  port: 8080 

What I saw on similar questions a suggestion to use async.eachOfLimit and that way to make a small amount of requests but for some it doesn't continue after the first iteration.
http://caolan.github.io/async/docs.html#eachOfLimit - The documentation if the module.

Simple code illustration:

router.post('/upload', function(req, res, next) {
.
.
.
    async.eachOfLimit(ids,100, CheckAndUpdate);
});

function CheckAndUpdate(DirtyId, callback){
   request({
     uri:  "http://XX.XX.XX.XXX:8080/rest/getTureIDs" + DirtyId, ",
     json: true
   }, function(err, res, data) {
     if (err) {
       console.log('Error:', err);
     } else if (res.statusCode !== 200) {
       console.log('Status:', res.statusCode);
     } else {
       if (data["TrueIDs"].length > 0){
         for (var index in data["TrueIDs"]){
           var fetchedID = data["TrueIDs"][index]["id"];
           UpdateSomethingForID(fetchedID);
         }
       }
     }
   });
}

function UpdateSomethingForID(ID, callback){
    request({
      uri:  "http://XX.XX.XX.XXX:8080/rest/SetTouchedToday/" + ID,
      json: true
    }, function(err, res, data) {
      if (err) {
        console.log('Error:', err);
      } else if (res.statusCode !== 200) {
        console.log('Status:', res.statusCode);
      } else {
        console.log("Great success");
      }
    });
  }

So the questions is:
How can I "spam" the request in a more proper way to avoid the ENOBUFS? Or what am I doing wrong with the eachOfLimit?

Thanks in advance!



via MaxBk

No comments:

Post a Comment