Apache Kafka wire format varint values are doubled

admin

When I look at the Kafka Protocol Guide, and inside that specifically at the Record format, it says that many number fields are encoded as varint. To explain varint, a link is given to Protobuf documentation. It’s a variable-length encoding; fair enough.

When I try to read the binary format of an Apache Kafka segment file (something akin to 0000000123.log), I am able to match the documented structures of the always fixed-length RecordBatch and the variable Records. However, the varint encoding does not seem to match the documentation. The resulting decoded value is always exactly double the actual number. Does anyone know, why that is?

Let’s make an example with a real file from my Apache Kafka 3.6 cluster.

On the left is the hexdump, and on the right I will annotate the meaning.

hexdump -C /var/lib/kafka/data-1/kafka-log0/perf-testing-0/00000000000000000007.log
[...]
00000290  76 61 6c 75 65 23 76 61  6c 75 65 23 31 31 00 00  |value#value#11..| (rest of previous batch, 15 bytes), New RecordBatch
000002a0  00 00 00 00 00 00 0d 00  00 01 83 00 00 03 6d 02  |..............m.| FirstOffset(..7 bytes), Length of batch(4 bytes), PartitionLeaderEpoch(4 bytes), Magic Byte/Message format v2 (1 byte)
000002b0  26 4b 85 d7 00 00 00 00  00 01 00 00 01 93 4d d0  |&K............M.| CRC(4 bytes), Attributes(2 bytes), LastOffsetDelta(4 bytes), FirstTimestamp(6 bytes..)
000002c0  5d c6 00 00 01 93 4d d0  5d c7 00 00 00 00 00 08  |].....M.].......| FirstTimestamp(..2 bytes), MaxTimestamp(8 bytes), ProducerId(6 bytes..)
000002d0  02 cc 00 00 00 00 00 01  00 00 00 02 cc 02 00 00  |................| ProducerId(..2 bytes), ProducerEpoch(2 bytes), FirstSequence(4 bytes), NumberOfRecords(4 bytes)=2    Record: Length(varint 2 bytes) cc 02, Attributes(1 byte)x, TimestampDelta(varlong 1 byte)
000002e0  00 30 6b 65 79 23 6b 65  79 23 6b 65 79 23 6b 65  |.0key#key#key#ke| OffsetDelta(varint 1 byte), KeyLength(varint 1 byte) 30, Key content (14 bytes..)
000002f0  79 23 6b 65 79 23 2d 31  31 31 8e 02 76 61 6c 75  |y#key#-111..valu| Key content (..10 bytes), ValueLength (varint 2 bytes) 8e 02, Value(remaining bytes from here on)
00000300  65 23 76 61 6c 75 65 23  76 61 6c 75 65 23 76 61  |e#value#value#va| Value content (16 bytes)
00000310  6c 75 65 23 76 61 6c 75  65 23 76 61 6c 75 65 23  |lue#value#value#| Value content (16 bytes)
[...]

In this snippet I have several varint length values, and two simple ones decode as follows:

Key length: 0x30 = 0|0110000 is equal to 110000 (bin) = 48 (dec) = 2 * 24
Value length 0x8e 0x02 = 1|0001110 0|0000010 combine to 100001110 (bin) =  270 (dec) = 2 * 135

When I manually count the number of bytes in the Key though (easy because it’s short) I see that the key is 24 bytes long. The decoded varint claims a length of 48.

Now let’s compare these values to the output of the kafka-dump-log.sh tool, which also confirms my observations:

bash /opt/kafka/bin/kafka-dump-log.sh --deep-iteration --print-data-log --max-bytes 1800 --files /var/lib/kafka/data-1/kafka-log0/perf-testing-0/00000000000000000007.log
[...]
baseOffset: 13 lastOffset: 14 count: 2 baseSequence: 1 lastSequence: 2 producerId: 525004 producerEpoch: 0 partitionLeaderEpoch: 877 isTransactional: false isControl: false deleteHorizonMs: OptionalLong.empty position: 671 CreateTime: 1732177321415 size: 399 magic: 2 compresscodec: none crc: 642483671 isvalid: true
| offset: 13 CreateTime: 1732177321414 keySize: 24 valueSize: 135 sequence: 1 headerKeys: []
[...]

It says it right there: keySize: 24 valueSize: 135. These sizes are only half of what the varints say.

Read more here: Source link