node.js – NGINX -> NodeJS 504 Timeout, NodeJs still process the request 2mins later

admin

So im getting a rare but annoying issue in my production application. Occasionally (1-2 request per week out of millions) will hang for 60s then NGINX will timeout. I added requestId to all my logging, and noticed that the request still gets processed… 2-3mins later NodeJs process the request, after NGINX has already closed the connection.

The request resolution time also seems reasonable ~153ms, about average for what a normal successful request should be.

NodeJS is not totally stalled in this time either, other requests get processed properly in this time.

To summarize:

  1. Nginx forwards the request to NodeJS
  2. Nginx waits 60s and times out
  3. After request has already timeout, 2mins later NodeJS picks up the request and processes it.

Any idea what could be going on?

Read more here: Source link