Worse RabbitMQ performance when doing less

I have RabbitMQ running in a Windows Docker container with Erlang (version 25.3) and RabbitMQ (version 3.11.18). This container operates in process isolation with four enabled schedulers. In my application, composed of services running in Docker containers, there are two modes: recording and non-recording. During recording, the system load is higher, and more data is transmitted through RabbitMQ. However, in non-recording mode, where the application performs fewer tasks and less data is send through RabbitMQ, RabbitMQ performance degrades. I confirmed this by running a monitoring process that sends a message with a timestamp, receives the message, and calculates the time difference between sending and receiving.

In recording mode, the time difference is approximately 10 ms, but in non-recording mode, the time differences are around 2000 ms at times (see graph 3 in the added image). I experimented with various settings for the Erlang runtime system, and although increasing the number of schedulers from 2 to 4 improved performance to some extent, the issue persists. What could be causing this behavior, and how can I enhance the performance?