Bug #2142585 “oslo.messaging might delete queues during RabbitMQ…” : Bugs : oslo.messaging

With the introduction of https://opendev.org/openstack/oslo.messaging/commit/8e3c523fd74257a78ceb384063f81db2e92a2ebd , queues might get deleted and not be recreated during RabbitMQ cluster issues that cause occasional request failures.

We experienced the following scenario with a 3 node RabbitMQ cluster:
1. oslo.messaging attempts to consume from a reply queue, but the amqp_ex.InternalError is thrown.
2. The queue was then successfully deleted.
3. A queue declare request is sent to recreate the queue, but this request fails and there is no retry logic, so the queue never gets recreated -> the Openstack request fails.

My suggestion is to remove this commit altogether since it goes against the recommendations of the RabbitMQ documentation:

“If a quorum of nodes cannot be recovered (say if 2 out of 3 RabbitMQ nodes are permanently lost) the queue is permanently unavailable and will need to be force deleted and recreated.”
https://www.rabbitmq.com/docs/4.2/quorum-queues#availability

If a queue has one or multiple noproc members, this does not mean that the nodes are permanently lost and you can usually remove and reintroduce their members by running:
rabbitmq-queues delete_member …
and
rabbitmq-queues add_member…

Afterwards the queues should be functional again. Since amqp_ex.InternalError does not necessarily mean that the queue is not recoverable, the health of queues should be ensured by using additional scripts that are outside the scope of oslo.messaging.

If we decide that the commit needs to be kept nonetheless, the declare request needs to be retried until the queue is successfully recreated.