google cloud platform – Define Schema for Bigquery Log Sink in GCP
I am using the Bigquery Log Sink for GCP. As stated by the documentation, BigQuery Log Sink defines the schema of the table based on the first record inserted.
Any changes in record structure in future will update the schema based on rules in this
link.
The problem we are facing is that we want to define the schema right from the first record insertion. We store both success and failure information in same record. Now till the time we do not encounter any failure record, the schema does not becomes complete.
Knowing the limitation provided by log sink, what can be the best ways to define schema upfront.
I am thinking of inserting one dummy record in the very start. Are there any better ways?
Read more here: Source link
