python – kafka consumer reads lot of duplicates

Im using python code to read the kafka stream message and send it to pubsub.
my python code triggered using cron job every minute.i can successfully read and write the message to bigquery. Now im getting lot of duplicates when i verify the data.
Below are my consumer properties.
consumer = KafkaConsumer (topicName, security_protocol=”PLAINTEXT”, sasl_mechanism=”PLAIN”,auto_offset_reset=”latest”, enable_auto_commit=True,ssl_check_host

do i missing something here.
Waiting for the experts suggestions.

Read more here: Source link