RabbitMQ Source connector in apache Flink is a bottleneck for our streaming app
We have a flink rmq source connector that is ingesting messages from a queue. We have been facing problems lately with ingestion during peak traffic hours. We have not seen this issue in the past.The size of the queue we are consuming off of balloons to over 1 million messages and in some cases close to 2 million messages before the queue size comes down as traffic dwindles. This leads us to believe the non parallel nature of rmq source connector is causing a bottleneck in our flink app. Unlike kafka which has partitions, I am unable to set the parallelism of the rmq source operator to more than 1 to parallelize ingestion. is there a solution to overcome this issue with rabbitmq source connector or its just a limitation with respect to flink’s rabbitmq source connector?
Below is the source code:
RMQConnectionConfig rmqConfig = new RMQConnectionConfig.Builder()
.setAutomaticRecovery(true)
.setPrefetchCount(17000)
.build();
RMQSource rmqSource = new RMQSource<>(
rmqConfig, // config for the RabbitMQ connection
rabbitQueue, // name of the RabbitMQ queue to consume
false, // use correlation ids; can be false if only at-least-once is required
new RMQCustomSchema() // custom deserialization schema
);
SingleOutputStreamOperator processedStream = env //stream execution environment
.addSource(rmqSource)
.name("RabbitMQ: " + rabbitQueue);
Note: We had reduced the prefetch count from 50000 to 17000 and reduced the checkpointing interval from 30 seconds to 10 seconds because it was overloading rabbitmq because it was not able to handle many ack’s at once. We had this issue last year. We are planning to reduce the prefetch from 17000 to 9500 and checkpointing interval from 10 seconds to 5 seconds to see if that does anything. Does that sound like a right approach in terms of resolving the bottleneck? If not, what other solutions can we use? We do not want to go through the route of having to implement our own rmq source connector to support parallelism since our rabbit queues support multiple consumers.
Thanks.
Read more here: Source link
