garbage collection – Application running out of memory due to apache kafka metric objects while using kafkaProducer

We try to send 1000s of messages to a kafka-topic using kafkaProducer(org.apache.kafka.clients.producer.KafkaProducer) objects on our tomcat application. Lately we have been observing an issue with large number of kafkaMetric objects being created on the heap.

We maintain the practise of closing kafkaProducer objects are a message is sent successfully. Also verified in our threaddumps, thats kafkaProducer objects are not hanging around.

But strangely JmxMBeanServer objects were present in heap space even after many days(as in they were not garbage collected too). Further these JmxMBeanServer objects consisted of metrics objects for kafka-producers present in a hashmap.

The hashmap keys strings were of the format key java.lang.String client-id=producer-322,type=producer-metrics

These hashmap objects were huge in number(more than 1000s).

Why were these metric objects not garbage-collected, even though the kafkaProducer’s objects were closed or shutdown.

Is there a way to stop auto creation of these metric objects.

PS:
kafka-clients jar version: 2.8.2

We tried several closing kafka producer objects at each send message operation, hoping metric objects would also be closed or garbage collected in few minutes. But they hogged up the heap.