Best Practice for Loading Data into Raw Layer in Azure Data Lake to SQL Pipeline (Truncate vs Append)

We’re using a layered architecture (Raw → Clean → Aggregate) to load data from Azure Data Lake Storage (ADLS) into Azure SQL using Azure Data Factory (ADF).

For the Raw layer, which is intended to store ingested data as-is from source files (e.g., CSV/Excel), I’m trying to decide between two loading strategies for each pipeline run:

Option 1: Truncate the Raw table before each load and insert the new file’s data.

Option 2: Append the new file’s data to the existing Raw table on every run.

Which of these approaches is recommended or considered best practice in a production data pipeline — especially with respect to auditability, reprocessing, and data traceability?

Any insights or architectural recommendations would be appreciated!

Read more here: Source link