Sunday, 7 May 2017

Send thousand dns request and save to csv - performance

I have problem with sending big amount of domains to check if they exist. I'm using node native dns and check in MX records.

I have array of 60k domains I send this to function that checking duplicated value and return unique domains, then I loop for each value and send to function: that verify domain if is valid, send dns request, check what MX received and return correct value, then I save it to csv file. Unfortunately it's saving to file only 1,2 to 1,5k domains not more. I don't received any error just stop saving. Maybe someone know what I doing wrong.

ev1.checkdomains(csvdomains, MXtocheck, function(uniquedata) {

var wstream = fs.createWriteStream(path);
for(i = 0; i < uniquedata.length; i++){

  var tmpdomian = uniquedata[i].toString();
  ev.searchMX(tmpdomian, arrMX, function(valid, domain, comment){

    if(valid){
          wstream.write(domain + ',' + comment + '\n');
         }

  });

}

also I think is important to noted that insite searchMX function I checking returned MX if maching array with MX servers that I don't want to have - that is not efficient because I'm using double loop, meaby this is the reason, but I don't know how to make this better.

dns.resolve(domain, 'MX', function(err, addresses) {    
  valid = false;
  if (err) {
   valid = false;
   comment = 'MX record not exist';   
  } 
  else if (addresses && addresses.length > 0) { 

  var tmp_comment = '';
  for(var i = 0; i < addresses.length; i++){
      for(var j = 0; j < arrMX.length; j++){
          if(addresses[i].exchange == arrMX[j]){
            valid = false;
            comment = 'is in wrong MX records ' + addresses[i].exchange;
            cb(valid, email, comment);
            return;
          }
      }
      tmp_comment += ' ' + addresses[i].exchange;
  }
   if(!valid){
     valid = true;
     comment = tmp_comment;
   }
  }
  else {
   valid = false;
   comment = 'error not specify'
  }
  cb(valid, email, comment);    
});

Thanks a million for any help to improve this code, or figure out what is wrong.



via cyprian

No comments:

Post a Comment