Azure SQL Hyperscale Partition Handling After ADF Copy from Parquet (Bronze Layer)
In our project, we are loading Parquet-partitioned data from the ADLS Gen2 Bronze layer into Azure SQL Hyperscale using Azure Data Factory Copy Activity (no data flow involved). This loading is part of a healthcare data migration involving ~80 TB historical + streaming data from on-prem DB2 to Azure SQL Hyperscale.
We have the following questions regarding partition behavior and optimization:
Partition Persistence Post-Loading After loading the Parquet-partitioned data into Azure SQL Hyperscale tables using Copy Activity, do the partition values (e.g., load_date, batch_id, etc.) persist in any usable form inside the destination table? Can we rely on these partition values to enable selective I/U/D processing from Kafka CDC later (e.g., for MERGE logic), without needing to re-merge in the ADLS staging zone?
Reflecting Partition Granularity in Hyperscale What is the recommended approach to reflect or preserve partition-level granularity in Hyperscale? For example, is there a way to materialize partition columns in Hyperscale schema, and then query data in partitioned batches (e.g., via WHERE batch_id = 'xyz') to optimize reconciliation and CDC post-processing?
Best Practice for Partition-Aware Merge Considering our MERGE logic is already applied in Databricks (before Hyperscale), is it a good idea to do partition-wise loading into staging tables in Hyperscale, and later move only the final records to main tables? Would this help in simplifying CDC conflict resolution and reduce query cost during retries or validation?
We look forward to your guidance on handling partitioned Parquet loads into Hyperscale and enabling optimal CDC and reconciliation.
Read more here: Source link
