apache zookeeper – How does Kafka store messages offsets on a local computer?
How does Kafka store messages on a local server or laptop?
I’m new to Kafka and just playing around with the tech for now but I’m curious to the answer because I started by looking at the Kafka Quickstart page to just start a Kafka service on my laptop. I sent several test messages and from some code, saw my topic had 100 messages.
Somewhere along the way, I read I can change where to store the logs in config/server.properties
under a setting log.dirs
. The default is set to /tmp/kafka-logs
so again, just to experiment, I shut off my Kafka service, recursive copied (cp -r) the dir to /home/username/kafka-logs
and restarted the service.
To my surprise, the offset was reset to 0. Shut off the Kafka server, went back to reset the log.dirs
setting to be /tmp/kafka-logs
, etc. etc. and when I checked, the offset was 100.
I had performed a recursive copy of the contents of the /tmp/kafka-logs dir so why did my laptop think the offset was 0? Why did I “lose” my messages? How does Zookeeper play into this and how do I check the contents of Zookeeper if it does?
Read more here: Source link