azure data factory – Incremental load for 3181 tables
I need to fastly run my incremental pipeline in azure data factory for 3181 tables.
I have 3181 tables in database(i.e azure SQL database )I have loaded these tables in datawarehouse(I.e azure SQL database). After that everyday data is coming in production for that purpose I have created incremental pipeline in adf so whatever the data has been coming into database will be incrementally loaded into datawarehouse tables. But whatever the pipeline I have created is taking long time like 12 hrs and more than 12 hrs . So I have segregated the tables in module level in pipeline . Each module may have 50tables ,some module have 100 tables etc .. so there are 51 modules for 3181 tables. So I have created 51 triggers for incremental pipeline based on parameter module and I have given one second gap for each schedule trigger . So there so much complexity in this . Is there any better approach
Read more here: Source link
