apache zookeeper – How does Kafka store messages offsets on a local computer?

How does Kafka store messages on a local server or laptop?

I’m new to Kafka and just playing around with the tech for now but I’m curious to the answer because I started by looking at the Kafka Quickstart page to just start a Kafka service on my laptop. I sent several test messages and from some code, saw my topic had 100 messages.

Somewhere along the way, I read I can change where to store the logs in config/server.properties under a setting log.dirs. The default is set to /tmp/kafka-logs so again, just to experiment, I shut off my Kafka service, recursive copied (cp -r) the dir to /home/username/kafka-logs and restarted the service.

To my surprise, the offset was reset to 0. Shut off the Kafka server, went back to reset the log.dirs setting to be /tmp/kafka-logs, etc. etc. and when I checked, the offset was 100.

I had performed a recursive copy of the contents of the /tmp/kafka-logs dir so why did my laptop think the offset was 0? Why did I “lose” my messages? How does Zookeeper play into this and how do I check the contents of Zookeeper if it does?

Read more here: Source link