apache kafka – Big delays between consuming latest messages
I have a Kafka topic with:
- 1 partition
- 1,000 messages
- 1,000,000 bytes each message
New messages produced with delay about 5 seconds, old messages regularry deleted.
When I consume messages from the tail of a topic, say last 5 messages, I receive it with random delay from 2 to 20 seconds between messages. I think this is way too slow. Is it normal delay for such a big messages in one partition and what can I do with it right now? I will try to break messages on small ones lately, but I can do it now. Can I configure something so it will consume fastely?
And I have 10 more topics on the same Linux server, but their messages not so big and consumed in not so slow manner. And this server is not so heavy loaded, top command shows about 20% memory consumption and 10% cpu consumption.
Read more here: Source link
