kafka + How to identify the specific corrupted topic partitions
We have encountered multiple instances of the following corrupted exception in our Kafka cluster, as shown in these examples:
[2025-08-13 02:05:11.011] ERROR [RaftManager id=1038] Had an error during log cleaning (org.apache.kafka.raft.KafkaRaftClient)
org.apache.kafka.common.errors.CorruptRecordException: Record size -1533948707 is less than the minimum record overhead (14)
[2025-08-12 23:52:53.053] ERROR [RaftManager id=1030] Had an error during log cleaning (org.apache.kafka.raft.KafkaRaftClient)
org.apache.kafka.common.errors.CorruptRecordException: Invalid magic found in record: 84
These exceptions appear in the server.log file.
The issue:
The server.log entries do not indicate which partition contains the corrupted logs.
What I want to ask:
I would like advice on whether it’s possible to identify the corrupted Kafka logs using the following approach:
bash kafka-dump-log.sh --print-data-log --files /kafka/kafka-log/topic-part-52/00000000000000000000.log | grep -i corrupt
The above is just an example. My goal is to create a script that runs over all Kafka log files under /kafka/kafka-log/, scans each partition log (e.g., topic-part-52/00000000000000000000.log), and detects which partitions contain corrupted logs.
Read more here: Source link
