apache spark – dataproc serverless slow consume kafka topic
I use dataproc serverless with java API to read kafka topic. The topic has only 2 partitions.
The topic receives 80 msg/sec.
After reading messages I repartition to 100, transform the data then write to BQ.
Since I repartition I see two stages one with 2 tasks to read data and a second one with 10 that transforms and writes the data.
When having
streaming.max.offsets.per.trigger=500
streaming.min.offsets.per.trigger=100
The task of reading is varying a lot in time between 7sec and 1min50.
While te task that transforms an dwrites data takes around 10 sec.
Any idea about why it’s taking too much time to read data.
Read more here: Source link

