apache kafka – What are the reasons java-clients/KafkaProducer was designed semi-blocking

Note: This is not an answer, it is a possible reason that I could think of, a subjective opinion

Refer: KafkaProducer below method

private Future<RecordMetadata> doSend(ProducerRecord<K, V> record, Callback callback)

One reason could be the producer is designed to operate under very high load. Sending of the records require a number of checks and prior tasks.

For example calculation of new partition if there is a new batch. If the producer sent the calling thread back immediately, it has 2 choices to do the tasks, spawn new threads for each send call that it receives or process serially the records from a queue (using a thread pool would lead to the same thing in a different way that there would be multiple serial paths).

Then to maintain high concurrency, the best option would be to use the calling thread to do the prior tasks, it reduces complexity of implementation and provides the performance as well.

Read more here: Source link