Some background:
What I do is receive a CSV file with some ids from the client. I run on the file and for each ID I make need to make a rest query that is followed by another rest query.
This thing works fine when I do it for 1,000 to ~ 5,000 ids but after that most of my responses at the logs are errors instead of "200".
The error is:
[Error: connect ENOBUFS XX.XX.XX.XXX:8080 - Local (undefined:undefined)]
code: 'ENOBUFS',
errno: 'ENOBUFS',
syscall: 'connect',
address: 'XX.XX.XX.XXX',
port: 8080
What I saw on similar questions a suggestion to use async.eachOfLimit
and that way to make a small amount of requests but for some it doesn't continue after the first iteration.
http://caolan.github.io/async/docs.html#eachOfLimit - The documentation if the module.
Simple code illustration:
router.post('/upload', function(req, res, next) {
.
.
.
async.eachOfLimit(ids,100, CheckAndUpdate);
});
function CheckAndUpdate(DirtyId, callback){
request({
uri: "http://XX.XX.XX.XXX:8080/rest/getTureIDs" + DirtyId, ",
json: true
}, function(err, res, data) {
if (err) {
console.log('Error:', err);
} else if (res.statusCode !== 200) {
console.log('Status:', res.statusCode);
} else {
if (data["TrueIDs"].length > 0){
for (var index in data["TrueIDs"]){
var fetchedID = data["TrueIDs"][index]["id"];
UpdateSomethingForID(fetchedID);
}
}
}
});
}
function UpdateSomethingForID(ID, callback){
request({
uri: "http://XX.XX.XX.XXX:8080/rest/SetTouchedToday/" + ID,
json: true
}, function(err, res, data) {
if (err) {
console.log('Error:', err);
} else if (res.statusCode !== 200) {
console.log('Status:', res.statusCode);
} else {
console.log("Great success");
}
});
}
So the questions is:
How can I "spam" the request in a more proper way to avoid the ENOBUFS? Or what am I doing wrong with the eachOfLimit
?
Thanks in advance!
via MaxBk
No comments:
Post a Comment