Saturday, 8 April 2017

Safest way to stop a node instance

I have a very busy web application:

  • Nginx as the "front", acting as a reverse proxy
  • Node server running on unprivileged ports, queried by Nginx

The application uses websockets (one socket open for each client). Nginx is configured in a very typical fashion:

server {
        listen X.Y.255.3:443 ssl http2;
        server_name example.com;

        client_max_body_size 0;


        ssl_certificate /etc/letsencrypt/live/example.com/fullchain.pem;
        ssl_trusted_certificate /etc/letsencrypt/live/example.com/chain.pem;
        ssl_certificate_key /etc/letsencrypt/live/example.com/privkey.pem;
        ssl_dhparam /etc/nginx/ssl/dhparam.pem;
        ssl_stapling on;
        ssl_stapling_verify on;

        location / {

          proxy_pass http://localhost:8082;
          proxy_http_version 1.1;
          proxy_set_header Upgrade $http_upgrade;
          proxy_set_header Connection 'upgrade';
          proxy_set_header Host $host;
          proxy_cache_bypass $http_upgrade;
        }
    }

My idea is that when I want to stop the server I do the following: I send a SIGTERM, so that: * I get the server to stop accepting connections * I cut off existing sockets (including websockets) * I clear up intervals and timeouts

At that point, the server should quit itself, since the event queue should be empty (once the last DB writes etc. are done).

I coded it this way:

        ...
        var server = http.createServer(app);

        process.on('SIGTERM', function(){
          server.close();
          console.log("TERMINATING THIS INSTANCE!");

          // This will make sure hotplate modules will get dir
          // of intervals and lose ends
          process.emit( 'hotplateShutdown');
          var handles = Array.prototype.slice.call(process._getActiveHandles())
          console.log( "Event queue has: ", handles.length );

          handles.forEach( (o ) => {  console.log( "Dealing with:", o.readyState ); o.unref && o.unref(); } );

        });

If after 10 seconds the process is still going, I send a SIGKILL (something is wrong at that point). However, note that I can re-run the server pretty much immediately, as after SIGTERM the server stops listening to the binding port. This means minimal downtime on restarts.

The websockets are killed as they should be.

Questions:

1) Is this a sane way to go about this? 2) If a DB call is half way, will it be completed? 3) Since I use express, I cannot really easily use this technique -- can I?

I noticed that the event queue contains 5 sockets. Those sockets are there even when Nginx is not running. I am not sure why 5, and I am not sure this is a sane way to go...

Comments/ideas?



via Merc

No comments:

Post a Comment