Load CSV to BQ with unknown columns
Hi, I quite often need to load CSV file from Cloud Storage to BigQuery with uknown number and type of columns. So in other words I need to create table in time of loading new csv. Is there any solution?
Hi, I quite often need to load CSV file from Cloud Storage to BigQuery with uknown number and type of columns. So in other words I need to create table in time of loading new csv. Is there any solution?
Hello Vit,
There’s currently no component to do this in Matillion but you can use the GCP CLI command inside the Bash Script component. The command would look similar to the one below.
#bq —location=[LOCATION] load —autodetect —source_format=[FORMAT] [DATASET].[TABLE] [PATH_TO_SOURCE]
bq —location=EU load —autodetect —source_format=CSV damian.matillion gs://mtln-damian/sample.csv
I hope this helps.
Best Regards,
Damian
To add an answer please login