Azure Event Hubs with Kafka, read_committed consumers stuck

Hi MALINGE Benjamin,

Thanks for the detailed context — your observations align closely with Kafka transactional behavior.

In Kafka, when consumers use isolation.level=read_committed, they can only read up to the last stable offset (LSO), which is the point just before the first open transaction. Any records beyond that remain invisible until the transaction is committed or aborted, which explains why records=0 is observed even though the highWatermark is higher.

Your values (offset=534, lastStableOffset=535, highWatermark=704) show the consumer is correctly stopping at an open transaction. Azure Event Hubs (Premium/Dedicated tiers) supports Kafka transactions, and the documentation states that expired transactions should be automatically aborted after the defined transaction.timeout.ms (default up to 15 minutes). Once the abort occurs, the LSO should advance and allow the consumer to continue.

Since the consumer remains blocked even after the timeout, this behavior is not expected and suggests the transaction was not closed as expected or that the abort marker did not advance the LSO. This points toward a potential service‑side issue rather than a client misconfiguration.

As next steps, please ensure your producer explicitly calls abortTransaction() in failure paths, keep transaction.timeout.ms appropriately low, and gather the transactional.id, partition information, timestamps, and producer/consumer logs for deeper investigation.

If you would like to provide feedback on this behavior, you can use the Azure feedback options available in the Azure portal or Event Hubs documentation pages.

If you need this investigated further, I recommend raising a support ticket with the details mentioned above so the engineering team can examine the transaction state on the service side.

Hope this helps. If you have any follow-up questions, please let me know. I would be happy to help.

Please do not forget to “Accept Answer” and “up-vote” wherever the information provided helps you, as this can be beneficial to other community members.

Read more here: Source link

Azure Event Hubs with Kafka, read_committed consumers stuck

Hi MALINGE Benjamin,

Thanks for the detailed context — your observations align closely with Kafka transactional behavior.

In Kafka, when consumers use isolation.level=read_committed, they can only read up to the last stable offset (LSO), which is the point just before the first open transaction. Any records beyond that remain invisible until the transaction is committed or aborted, which explains why records=0 is observed even though the highWatermark is higher.

Your values (offset=534, lastStableOffset=535, highWatermark=704) show the consumer is correctly stopping at an open transaction. Azure Event Hubs (Premium/Dedicated tiers) supports Kafka transactions, and the documentation states that expired transactions should be automatically aborted after the defined transaction.timeout.ms (default up to 15 minutes). Once the abort occurs, the LSO should advance and allow the consumer to continue.

Since the consumer remains blocked even after the timeout, this behavior is not expected and suggests the transaction was not closed as expected or that the abort marker did not advance the LSO. This points toward a potential service‑side issue rather than a client misconfiguration.

As next steps, please ensure your producer explicitly calls abortTransaction() in failure paths, keep transaction.timeout.ms appropriately low, and gather the transactional.id, partition information, timestamps, and producer/consumer logs for deeper investigation.

If you would like to provide feedback on this behavior, you can use the Azure feedback options available in the Azure portal or Event Hubs documentation pages.

If you need this investigated further, I recommend raising a support ticket with the details mentioned above so the engineering team can examine the transaction state on the service side.

Hope this helps. If you have any follow-up questions, please let me know. I would be happy to help.

Please do not forget to “Accept Answer” and “up-vote” wherever the information provided helps you, as this can be beneficial to other community members.

Read more here: Source link

Azure Event Hubs with Kafka, read_committed consumers stuck

Hi MALINGE Benjamin,

Thanks for the detailed context — your observations align closely with Kafka transactional behavior.

In Kafka, when consumers use isolation.level=read_committed, they can only read up to the last stable offset (LSO), which is the point just before the first open transaction. Any records beyond that remain invisible until the transaction is committed or aborted, which explains why records=0 is observed even though the highWatermark is higher.

Your values (offset=534, lastStableOffset=535, highWatermark=704) show the consumer is correctly stopping at an open transaction. Azure Event Hubs (Premium/Dedicated tiers) supports Kafka transactions, and the documentation states that expired transactions should be automatically aborted after the defined transaction.timeout.ms (default up to 15 minutes). Once the abort occurs, the LSO should advance and allow the consumer to continue.

Since the consumer remains blocked even after the timeout, this behavior is not expected and suggests the transaction was not closed as expected or that the abort marker did not advance the LSO. This points toward a potential service‑side issue rather than a client misconfiguration.

As next steps, please ensure your producer explicitly calls abortTransaction() in failure paths, keep transaction.timeout.ms appropriately low, and gather the transactional.id, partition information, timestamps, and producer/consumer logs for deeper investigation.

If you would like to provide feedback on this behavior, you can use the Azure feedback options available in the Azure portal or Event Hubs documentation pages.

If you need this investigated further, I recommend raising a support ticket with the details mentioned above so the engineering team can examine the transaction state on the service side.

Hope this helps. If you have any follow-up questions, please let me know. I would be happy to help.

Please do not forget to “Accept Answer” and “up-vote” wherever the information provided helps you, as this can be beneficial to other community members.

Read more here: Source link