google cloud platform – Handling schema evolution (nested column) – BigQuery External tables

I have a GCS bucket which contains Avro files. On top of this, resides a BigQuery External table.

The schema could change (as always). A new field could be added or a field could be removed, or datatype of a field could change in a nested column

As of now, how we are handling this schema evolution is, creating a new folder during every schema change. Reason for that is when both schemas coexist in the same folder, we see that old schema overrides new one i.e. new fields in nested column have null values for both old and new schemas, which is correct for old schema but not for the new schema as files with new schema have the new field populated.

How do I make sure that the values are populated accordingly in BigQuery and there is no need for creating a new folder in the gcs bucket whenever schema is evolved.