Friday 9 June 2017

Kue.js - blocked background jobs

I am using Sails.js, which connects to Kue.js background jobs. My Kue.js sometimes blocks every request and i have no idea why.. If it blocks, then it is not possible to connect to Kue in any possible way. The only way is to restart kue background jobs. And in production i dont really wanna do that so often.

This is my package.json for Kue.js background jobs:

{
  "name": "background_jobs",
  "version": "1.0.0",
  "description": "description",
  "main": "kue_worker.js",
  "dependencies": {
    "cluster": "^0.7.7",
    "os": "^0.1.1",
    "underscore": "git://github.com/jashkenas/underscore.git",
    "kue": "~0.11.0",
    "request": "^2.81.0",
    "pg": "~6.1.5",
    "pg-promise": "~5.6.4",
    "kue-scheduler": "0.7.0"
  },
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "author": "",
  "license": "ISC"
}

This is my kue_worker.js file, where is background-jobs logic

var _ = require('underscore');
var kue_scheduler = require('kue-scheduler');
var Queue = kue_scheduler.createQueue();
var pgp = require('pg-promise')(/*options*/);
var cn = {
    host: 'localhost', // server name or IP address;
    port: 5432,
    database: 'development',
    user: 'hidden',
    password: 'hidden'
};
var db = pgp(cn);
var kue = require('kue'), cluster = require('cluster'), queue = kue.createQueue();
var clusterWorkerSize = require('os').cpus().length;
var sails_url = 'http://localhost:1337/';

if (cluster.isMaster) {
  kue.app.listen(3000);
  for (var i = 0; i < clusterWorkerSize; i++) {
    cluster.fork();
  }
  console.log('started');
} else {

  queue.process('preloader', 10, function(job, done){

        console.log('started calculation for ', job.data);
        // all the calculation logic
  });
}

And this is the request i make (which normally always forces to start this background job)

curl -H "Content-Type: application/json" -X POST -d \
    '{
       "type": "preloader",
       "data": 77,
       "options" : {
         "attempts": 1,
         "priority": "normal"
       }
     }' http://localhost:3000/job

This is what my redis console shows (this is the same all the time on every machine i use it, so probably its not important, i see this one message and no other logs from redis):

3827:M 09 Jun 08:54:17.032 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
3827:M 09 Jun 08:54:17.032 # Server started, Redis version 3.0.6
3827:M 09 Jun 08:54:17.032 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
3827:M 09 Jun 08:54:17.042 * DB loaded from disk: 0.010 seconds
3827:M 09 Jun 08:54:17.042 * The server is now ready to accept connections on port 6379

I have very CPU-heavy background jobs, where one process lasts sometimes for 1 minute.

Right now i was able to catch this Kue requests block - after ~50 requests to Kue in 3 hours.

I have tried to restart my Sails.js server, but it does everything OK - it does send the request, and gets no errors, but my Kue does not get anything.. Otherwise i would see a messages in my Kue.js background jobs console, that it gained a request.

I was also trying to make the same request with curl, but its the same - no message in Kue worker and no result overall - it should update my database.

Please help me to locate the problem, i am totally confused and i have no more ideas how to solve it.



via mansim

No comments:

Post a Comment