Sunday, 16 April 2017

how many request can logstash (or instance can handle)

I am setting up log for nodejs. The flow is like this, using buyan lib to print log to the console and send to logstash as the sametime.

var log = bunyan.createLogger({
  name: 'example',
  serializers: bunyan.stdSerializers,
  streams: [{
      level: 'debug',
      stream: process.stdout
  },{
      level: 'debug',
      type: "raw",
      stream: bunyantcp.createStream({
          host: 'logstash',
          port: 5000,
          max_connect_retries: -1, // Don't give up on reconnecting
          retry_interval: 1000 
      })
  }],
  level: 'debug'
}); 


function doSetTimeout(i) {
  setTimeout(function() { 
    log.debug(i)
    }, 1000);
}

for (var i = 1; i <= 1000000; ++i)
  doSetTimeout(i);

The thing is when I set i = 10000, logstash can handle all of them. But if I switched i to 1000000. It just processed 4907 logs.

That's it, I don't know where exactly the problem is. Is it beacause of the instance's bandwidth or the logstash its self or even the machine running the script is not quality enough?



via Huy Hoang

No comments:

Post a Comment