node.js – Nodejs RabbitMQ Ack not received by RabbitMQ Server

We have 3 Node RabbitMQ cluster. All queues are created by durable & HA_ALL ( Mirroring enabled when I hover over the Node information of each queue).

Issue: We have set of consumers which are processing the messages from queues.

Now from yesterday we have been running to problem, consumer get the message processes them and also call ack function using amqplib & We can also confirm ack didn’t throw any error. But the ack msg is not getting removed from queue. In fact to put it in plain words, consumer sent ack to server without no errors, but somehow the ack is not received by the server and keeps waiting for ack from consumer. When it enters this state all queues across all consumers gradually seems to have the same problem.

RabbitMQ logs does not indicate any issues, to confirm ack is getting success.. We have added logs post ack() function with try catch..

When this issue happens, if restart all of my consumers, without any issues it’s getting processed for certain period of time and goes to issue state. ( This indicates there is no issue with consumers processing and ack function & also does not seem to be rogue payload causing nack ).

We also looked for double ack errors in log 406.. Not able find any.