python – Tasks stuck in broker

I have RabbitMQ on a Docker container with result_backed to PostgreSQL (if local it uses SQLite). When running in multiprocessing mode, tasks get stuck in RabbitMQ. Why is that?

If I run Celery with regular pool implementation (multiprocessing), tasks do not return any result (if I don’t set timeout it never ends). When I look at the RabbitMQ management console I see that they are stuck in unacked. If I close the Celery app and then reopen it, all those stuck tasks are recalled (task … received), but they are still stuck.

If I start the app again with -P eventlet or threads, all of the stuck tasks are recalled and resolved (and in RabbitMQ all the unacked messages become ready). Also, when running in multiprocess mode I see no entry in the backend, but when changing to multithreading I see new entries for those tasks called while Celery was in multiprocessing mode.

The task itself is add(x, y): return x+y.

Read more here: Source link