Kafka consumer – how to recognized offset skipping/missing offsets?

setting auto.offset.reset to none for the consumer and expected the consumer to stop in case it can’t find an offset

That’s what it’ll do, yes.

In practice it immediately throws an exception when the consumer gets instantiated because there’s no first/initial offset

You’ll need to actually initialize the group first, then seek it to the earliest offset. E.g. kafka-consumer-offsets --reset-offsets --to-earliest --group connect-<name>

Something like “throw exception if offset is missing/skipped, but not on first start”?)

There’s nothing to differentiate auto.offset.reset between “first” and “next” starts. But, you could create the connector with consumer.override.auto.offset.reset=earliest, then wait for it to be running, then set it back to none with a PUT /config call. Then repeat whenever it stops running again.

JMX metric we could monitor in case a consumer is skipping messages

Not that I know of; the metrics are mostly reporting bytes processed. You’d have to additionally track how many bytes you expect it to read.

You’d need other monitoring solutions to detect log segments being deleted on the broker, and tracking those offset ranges compared to the offsets your consumer is currently reading.