Real-time Reconciliation Between Live DB2 and Azure SQL in CDC Streaming

Question:

We’re ingesting CDC data from IBM DB2 into Azure SQL Hyperscale via Kafka + Databricks Structured Streaming. Our challenge is reconciling live DB2 tables with their corresponding Hyperscale rows.
Since DB2 is a live system, a row may be updated multiple times while its CDC event is still in-flight to Hyperscale. Hence, real-time value comparison is non-deterministic.
Our question is:
What is Microsoft’s recommended approach to perform accurate row-level or hash-based reconciliation in such real-time CDC use cases?
Is it advisable to use commit timestamp ranges (JRN_TIMESTAMP) from the CDC payload to snapshot-filter DB2 source rows for matching?
Are there any built-in tools or connectors (in ADF or Azure SQL) that support commit-aware reconciliation logic?
In the absence of full commit sync, is it acceptable to reconcile only between CDC payload vs Hyperscale, skipping DB2 for real-time verification?
What are the known limitations of reconciling a live source like DB2 against a downstream system like Hyperscale, and how can they be mitigated?

Read more here: Source link