Load CSV to BQ with unknown columns
Hi, I quite often need to load CSV file from Cloud Storage to BigQuery with uknown number and type of columns. So in other words I need to create table in time of loading new csv. Is there any solution?
1 Community Answers
Matillion Agent
Damian Chan —
Hello Vit,
There’s currently no component to do this in Matillion but you can use the GCP CLI command inside the Bash Script component. The command would look similar to the one below.
#bq —location=[LOCATION] load —autodetect —source_format=[FORMAT] [DATASET].[TABLE] [PATH_TO_SOURCE]
bq —location=EU load —autodetect —source_format=CSV damian.matillion gs://mtln-damian/sample.csv
I hope this helps.
Best Regards,
Damian
Post Your Community Answer
To add an answer please login