data engineering – Acks in Apache Kafka

By default, the acknowledgement setting is set to “1” in Kafka, which means the producer will receive acknowledgement after the leader broker has received the message. This setting is known as “acks=1” or “acks=leader”.

The reason behind this default setting is to strike a balance between reliability and performance. When the producer receives acknowledgement from the leader broker, it knows that at least one broker has successfully received the message. This provides a certain level of reliability without sacrificing too much performance.

Setting the acknowledgement to “all” (acks=all) would require acknowledgement from all in-sync replicas (ISRs) before considering the message as successfully sent. This setting offers the highest level of reliability because it ensures that the message has been replicated to all brokers in the ISR. However, it comes at the cost of increased latency and potential performance impact.

Choosing the appropriate acknowledgement setting depends on the specific use case and the desired trade-off between reliability and performance. If data loss is unacceptable and durability is the utmost priority, setting acks to “all” would be appropriate. On the other hand, if performance is critical and a small chance of data loss is acceptable, the default setting of “1” provides a good balance.