Dataflow: Latency in ‘Kafka to table in CloudSQL d…

Hi.

I have a Dataflow pipeline (in Java) which reads messages from Kafka topic and writes row updates to CloudSQL (Postgres).

The pipeline doesn’t perform any aggregations and transformation steps take 4-5 milliseconds. According to metrics, reading of Kafka record from topic takes 400-500 ms. But Dataflow shows that processing latency is 8-10 seconds. I can’t understand where time is spent. How to achieve sub-5-second latency ?

For JDBC write I use batching and sharding (JdbcIO.write().withBatchSize().withAutoSharding().Also, connection pooling is enabled, CloudSQL instance has enough resources.