Saturday, 13 May 2017

nginx tcp SYN packet not receiving ACK

My server setup is nginx directly connects to a node.js server (port 8000). The symptom is sometimes there are some 504 logs in nginx log. And node.js log doesn't show any sign of ever receiving the request. I then enabled tcp log using iptables, which logs all tcp packets to port 8000. After checking the tcp log, it seems that nginx was trying to establish a tcp connection with node.js server, but it never succeeds. It just kept retrying sending SYN packets and then got timed out by nginx. Here's an example (tcp + nginx log):

13:44:44 sp:48103 dp:8000 SYN
13:44:45 sp:48103 dp:8000 SYN
13:44:47 sp:48103 dp:8000 SYN
13:44:51 sp:48103 dp:8000 SYN
13:44:59 sp:48103 dp:8000 SYN
13:45:15 sp:48103 dp:8000 SYN
13:45:44 nginx 504

During the period, the CPU load is pretty light, memory < 50%, incoming request is less than 50 per minute. And other requests were processed normally.

Server is Ubuntu 14.04.2 LTS

Any idea what's going on? Seems like an OS level issue? Thank you in advance.



via Guoxing Li

No comments:

Post a Comment