Solved: How to stream BigQuery table in Node.js?

You can stream a BigQuery table bit by bit using the BigQuery.Streaming.createQueryStream method. This method takes a query and a destination table as arguments. The query can be any valid BigQuery query, and the destination table must be a table that is configured for streaming.

When you create a streaming query, BigQuery will automatically start streaming the data from the source table to the destination table. The data will be streamed in batches, and each batch will contain a subset of the data from the source table.

You can use the BigQuery.Streaming.on event handler to listen for data being streamed to the destination table. The data event handler will be called for each batch of data that is streamed.

Here is an example of how to stream a BigQuery table bit by bit:

const {BigQuery} = require('@google-cloud/bigquery');
const bigquery = new BigQuery();

// Create a streaming query for the `publicdata.samples.github_nested` table.
const query = 'SELECT url FROM `publicdata.samples.github_nested`';
const destinationTable="my-streaming-table";

// Create a streaming query stream.
const stream = bigquery.createQueryStream(query, destinationTable);

// Listen for data being streamed to the destination table.
stream.on('data', function(row) {
// Do something with the data.
});

// Stop the streaming query.
stream.on('end', function() {
// The streaming query has stopped.
});

This code will create a streaming query for the publicdata.samples.github_nested table and stream the data to the my-streaming-table table. The data will be streamed in batches, and each batch will contain a subset of the data from the source table. You can use the data event handler to listen for data being streamed to the destination table.

I would love to hear your thoughts!

 

Roderick G
Community Manager
Google Cloud Community

Read more here: Source link