java – Sudden spike of Spring Boot RabbitMQ consumers on one server
We are using spring boot version 3.3.1
with RabbitMQ version 3.8.19
(planning on upgrading to the next version soon). We have the following configuration for RabbitMQ and Spring Boot App.
RabbitMQ
- Currently on version
3.8.19
. - 16 lazy mirror queques.
- Dedicated DLQs for all.
Spring Boot
- Currently on version
3.3.1
. - 4 app servers in cluster.
- Using
250
(default) prefetch value. - minConsumer are 2 and maxConsumer are 50 and are of type
Simple
.
So the issue we have is during a small high volume (2000-3000) of messages to RabbitMQ, one the spring boot app spikes its consumers to maxConsumer, while the rest of 3 are just chilling with the minConsumer. We are seeing that in RabbitMQ management console as we have a custom tag strategy configuration on spring boot for RabbitMQ. After which the app with all those consumers sits there with those messages as UNACK
and takes >30
mins to process them and ACK back to RabbitMQ, which usually takes only few mili. We checked logs and did not see any errors or even slowness in our external services (database, apis, etc). Not sure but if what could be the issue, but we have had the same app and rabbit configurations for years and this started happening suddenly. Any help to resolve this issue would be appreciated.
Read more here: Source link