apache zookeeper – need to run kafka-storage.sh on every broker and controller

Basically the title.

We’ve been running kafka with zookeeper for years. Setup was to form Zookeeper cluster and then let brokers connect to it. Everything in puppet. No extra steps and after some restarts cluster eventually got up.

Now we’re migrating to Kraft (in-place) and I noticed I need to format the storage on every controller using kafka-storage.sh utility. I thought it’s only part of Kraft migration process but apparently this is needed even when new Kraft clusters are built from scratch. As per kafka.apache.org/documentation/#quickstart

I guess I need to swallow my ‘why they couldn’t just automate it as part of boot up process’ but I have some follow up questions I can’t find answers to:

Do I need to generate the UUID once and then format the storage on each Kraft controller node separately using same UUID?
Do I need to do the same on every broker or that is somewhat automated?
When it comes to Zookeeper -> Kraft migration I need to do it only on new controllers?: docs.confluent.io/platform/current/installation/migrate-zk-kraft.html#step-3-format-storage-with-the-id-you-saved-previously