Name is required.
Email address is required.
Invalid email address
Answer is required.
Exceeding max length of 5KB

Load CSV to BQ with unknown columns

Hi, I quite often need to load CSV file from Cloud Storage to BigQuery with uknown number and type of columns. So in other words I need to create table in time of loading new csv. Is there any solution?

1 Community Answers

Matillion Agent  

Damian Chan —

Hello Vit,

There’s currently no component to do this in Matillion but you can use the GCP CLI command inside the Bash Script component. The command would look similar to the one below.

#bq —location=[LOCATION] load —autodetect —source_format=[FORMAT] [DATASET].[TABLE] [PATH_TO_SOURCE]

bq —location=EU load —autodetect —source_format=CSV damian.matillion gs://mtln-damian/sample.csv

I hope this helps.

Best Regards,
Damian

Post Your Community Answer

To add an answer please login