Kafka Stream DSL groupby() vs Processor API performance tradeoff
Kafka Streams DSL groupBy() operation can lead to repartitioning based on new key chosen
In this, skewing or possibility of repartitioning data to single partition is possible
This is significant performance bottleneck for usecase of counting messages for a day
Trying Processor API with a KeyValueStore (Persistent)
Where Record Key and Value both can be accessed and key can be ch
anged
And context.forward allows to forward this changed record out from process()
This seems much performant – as it doesn’t involve repartition topic creation
What is drawback of this usecase?, Is it not resilient – because of state wipe out? And it’s not being backed by a changelog topic?
Please suggest?
Read more here: Source link
