google cloud platform – BigQuery high cost when refreshing a materialized view
I Have two partitioned tables, let’s say t1 and t2, in BQ that are being refreshed once a day.
These two tables are very used for KPIS calculation and so it was decided to create a partitioned Materialized View lets call it MV with data from t1 and t2.
The query in MV is basically doing some calculations with data from t1 and t2 which are joined by date.
When I run manually every day the refresh of the materialized view after day load,
CALL BQ.REFRESH_MATERIALIZED_VIEW('project.dataset.mv')
it seems that process more data than expected, for example:
Whether I execute the query embedded inside the MV for the daily day only it process 6GB of data, but, If I run the CALL BQ.REFRESH_MATERIALIZED_VIEW it process up to 20TB.
- Is there a way to refresh only one day of data (incremental refresh) into the materialized view?
There is also another curious situation in which when I check the INFORMATION_SCHEMA.JOBS_BY_PROJECT to see how much it processed and the cost of it, I see that for the same CALL, there are two jobs:
1. One starting with bquxjob
2. Another one starting with script_job_
Both of them processing same amount of data.
- Is there also a way to avoid this? or this is something expected and why?
I expect to get lower charges when executing the refresh.
Read more here: Source link
